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Abstract. We study general (not necessarily Hamiltonian) first-order symmetric sys- 
tems Jy'{t) — B{t)y{t) = A{t)f{t) on an interval I = [a, b) with the regular endpoint a. 
It is assumed that the deficiency indices n±{T^i^) of the minimal relation Tj^in in L^(X) 
satisfy n—{T^i^) < n+{T^i^). By using a Nevanlinna boundary parameter r = t(A) at 
the singular endpoint b we define self-adjoint and A-depending Nevanlinna boundary con- 
ditions which are analogs of separated self-adjoint boundary conditions for Hamiltonian 
systems. With a boundary value problem involving such conditions we associate the m- 
function m{-), which is an analog of the Titchmarsh-Weyl coefficient for the Hamiltonian 
system. By using m- function we obtain the Fourier transform V : L^(X) — )■ with 
the spectral function S(.) of the minimally possible dimension. If V is an isometry, then 
the (exit space) self-adjoint extension T of Tj^in induced by the boundary problem is 
unitarily equivalent to the multiplication operator in L'^{E); hence the spectrum of T 
is defined by the spectral function We show that all the objects of the boundary 

problem are determined by the parameter t, which enables us to parametrize all spectral 
function E(-) immediately in terms of r. Similar results for various classes of boundary 
problems were obtained by Kac and Krein, Fulton , Hinton and Shaw and other authors. 



1. Introduction 
Let H and H be finite dimensional Hilbcrt spaces and let 

(1.1) Ho-~H®H, m:=Ho®H ^H®H®H. 

The main object of the paper is first-order symmetric system of differential equations 
defined on an interval T — [a, &),— oo < a < b < oo, with the regular endpoint a and regular 
or singular endpoint b. Such a system is of the form [3, 15] 

(1.2) Jy'{t)~B{t)yit)=A{t)fit), tGl, 
where B{t) — B*{t) and A(t) > are the [HJ-valued functions on X and 

Throughout the paper we assume that the system (1.2) is definite. The latter means that 
for any A G C each common solution of the equations 

(1.4) Jy'{t) - B{t)y{t) = AA(t)y(t) 

and A{t)y{t) = (a.e. on I) is trivial, i.e., y{t) — 0, t ^ I. 
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System (1.2) is called Hamiltonian system if iJ = {0}. In this case one has 




: H®H ^ H®H. 



In what follows we denote by L\{X) the Hilbert space of H- valued Borel measurable 

functions /(•) (in fact, equivalence classes) on I satisfying ||/||^ := / (A(t)/(t), /(t))H dt < 

I 

00. 

Investigations of symmetric systems is motivated by several reasons. For instance, systems 
(1.4) form more general objet than formally self-adjoint differential equation of arbitrary 
order with matrix coefficients. Such equation is reduced to a system of the form (1.4) with 
J given by (1.3) (see [28]). Emphasize that presence of the term ilg in (1.3) under this 
reduction characterizes odd order equations, although even order equations are reduced to 
Hamiltonian systems (with J given by (1.5)). Moreover, the Krein-Feller string equation is 
also reduced to Hamiltonian system (1.4) ([15, Chapter 6, §8]). 

As it is known, the extension theory of symmetric linear relations gives a natural frame- 
work for investigation of the boundary value problems for symmetric systems (see [4, 11, 
12, 18, 25, 32, 33, 44] and references therein). According to [25, 33, 44] the system (1.2) 
generates the minimal linear relation Tmin and the maximal linear relation Tmax in L\{I). 
It turns out that T,ni,i is a closed symmetric relation with not necessarily equal deficiency 
indices n±(Tmin)- Since system (1.2) is assumed to be definite, n±(Tmin) can be defined as 
a number of L^-solutions of (1.4) for A e C±. Moreover, Tmax = T^in ^^d the equality 

(1.6) [y,z]b = \\m{Jy{t),z{t)), y, ^ e domT^ax, 

defines a skew-Hermitian bilinear form on the domain of Tmax- 

A description of various classes of extensions of Tmin (self-adjoint, m-dissipative, etc.) in 
terms of boundary conditions is an important problem in the spectral theory of symmetric 
systems. Assume that the system (1.2) is Hamiltonian and n+(Tmin) = '^-(Tmin)- Let 
y{t) = {2/o(i)j 2/i(i)}(G H (B H) be the representation of a function y G domT^ax- Then 
according to [20] the general form of self-adjoint separated boundary conditions is 

(1.7) cosBi j/o(a) + sinSi ?yi(a) = 0, [v,Xj]b = ^, i G {1, ... , z^fa}, ?y G domTInax, 

where Bi is a self-adjoint operator on H and {XjliN^b = "±(Tmin) — dimJJ, is a certain 
system of functions from domTmax- The vector y^ := {[y, Xj\b}'i "= C'''' is called a singular 
boundary value of a function y G domTTnax- Observe that for ordinary differential operators 
description (1.7) goes back to I.M. Glazman (see [1, Appendix 2, §5]), while the form of the 
boundary conditions (at regular endpoints) goes back to F.S. Rofe-Beketov [45]. Note also 
that the notion of a singular boundary value can be found in the book [13, Ch.13.2]). 

Boundary conditions (1.7) generate a self-adjoint extension A of Tmin given by A = 
{{y, /} G Tmax : y satisfies (1-7)}. The resolvent {A — \)~^ of A is defined as follows: for 
any / G L\il) vector y = [A — \)~^ f \s the L^-solution of the equation 

(1.8) Jy'(i)-B(%(i) = AA(t)y + A(i)/(i), / G Li(I), AgC\M, 

subject to the boundary conditions (1.7). Moreover, according to [20] the Titchmarsh - Weyl 
coefficient Mtw(A)(g [H]) of the boundary problem (1.8), (1.7) is defined by the relations 

v(t,\) ip{t,x)MTwW + i^(t,\) G c\[H,m], 

[v{-,X)h,Xj]b = 0, hGH, j G {1, i^b}. 
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Here and ■^(•, A) are the [H,M ® HI]-valued operator solutions of Eq. (1.4) with the 

initial data 

ip{a,X) = (sinSi, — cosBi)^ and tp{a, X) — (— cosBi, siniJi)^. 

Note also the papers [27, 30], where the Titchmarsh - Weyl coefficient for Hamiltonian 
systems is defined in another way. By using Mtw{') one obtains the Fourier transform with 
the spectral function S(-) of the minimally possible dimension N-^ = dimH (see [11, 12, 22, 
25]). 

It turns out that for general (not necessarily Hamiltonian) symmetric systems the situa- 
tion is more complicated. In particular, it was shown in [42] that non- Hamiltonian system 
(1.2) does not admit separated self-adjoint boundary conditions. Moreover, the inequality 
n+(Tmin) 7^ '^-(Tmin), and hence absence of self-adjoint boundary conditions is a typical sit- 
uation for such systems. For instance, in the limit point case at h one has n+{T^i^) = dimi? 
and n_(Tniin) — A\m H -\- dim. H . Such circumstances make it natural to investigate the fol- 
lowing problems: 

• To find (might be A-depcnding) analogs of self-adjoint separated boundary conditions 

for general systems (1.2) and describe such type conditions; 

• To describe in terms of boundary conditions all spectral matrix functions that have 
the minimally possible dimension and investigate the corresponding Fourier transforms. 

In the paper we solve these problems for symmetric systems (1.2) assuming that (Tmin) < 
n-|_(Ti„in)- However to simplify presentation we assume within this section that n_(Ti„in) = 
n+(rmin) (the case n+(Tmin) < n_(Tmin) will be treated elsewhere). We first show that 
there exists a finite-dimensional Hilbert space %>) and a surjective linear mapping 

Fb = (ro6, Fb, Tibf ■■ dom 

such that the bilinear form (1.6) admits the representation 

[y,z\b = (Fo6y,ri62;) - (J?iby-,'^obz) + i(Xby,'^bz), y,z G domTmax- 

It turns out that F^y can be chosen in the form of a singular boundary value of y e dom T^ax 
(see Remark 3.5). Moreover, each proper extension of Tmin can be defined by means of 
boundary conditions imposed on vectors y{a) = {yo{a), y{a), yi(a)}(€ H ® H ® H) and 
T^y = {Fo^y, T^y, Fitj/}(e Hb ® H ® Hb)- In particular, a linear relation T given by 

T := {{y, /} e T^a. : 2/i(a) = 0, y(a) = F^y, Fo^y = Fi^y = 0}, 

is a symmetric extension of Tmin and plays a crucial role in our considerations. 
Recall that a generalized resolvent of T is an operator- valued function given by 

RW = PlI(i){T - A)-' r Ll{I), A e C \ M, 

where T is an exit space self-adjoint extension of T acting in a wider Hilbert space D 
L\{I). Moreover, the spectral function of T is defined by 

F{t) = PLl{i)E{t) \ L\{X), t e K, 

where E{-) is the orthogonal spectral function (resolution of identity) of T. We show that 
each generalized resolvent y = R{X)f, f € L\{I), is given as the Z/^(I)-solution of the 
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following boundary-value problem with A-depending boundary conditions: 

(1.9) Jy' -B{t)y = XA(t)y + A{t)f{t), t&X, 

(1.10) yi{a) = 0, y{a)=ft,y, 

(1.11) Co(A)ro6y + Ci(A)rifc?y = 0, A e C \ M. 

HereCo(-) andCi(-) are the components of a Nevanlinna operator pair t(-) = {{Co{-),Ci{-))} 
with values in [Hb] ® [Hb], so that formula (l-H) defines a Nevanlinna boundary condition 
at the singular cndpoint b. One may consider a pair r = t(-) as a boundary parameter, 
since i?(A) runs over the set of all generalized resolvents of T when r runs over the set of 
all Nevanlinna operator pairs. To indicate this fact explicitly we write R{\) = RtW and 
F{t) = Fr{t) for the generalized resolvents and spectral functions of T respectively. More- 
over, we denote by T = T"^ the exit space self-adjoint extension of T generating Rt{-) and 

Fr{-). 

The boundary value problem (1.9)-(1.11) defines a canonical resolvent Rt{X) if and only 
if T is a self-adjoint operator pair r = {(cos sin S)} with some B = B* £ [Hb]- In this 
case Rt{X) = (T^ - A)~^, A e C \ M, where T'^ is a self-adjoint extension of T in L\{I) 
defined by the following mixed boundary conditions : 

(1.12) = {{?/,/} eT^ax:yi (a) = 0, y{a) = fby, cosB • Tobj/ + sinS • Fuj/ = 0}. 

For Hamiltonian systems the equalities in the right-hand side of (1.12) take the form of 
self-adjoint separated boundary conditions 

(1.13) j;i(a)=0, cosB-Foby + sinB-Fib?y = 0. 

Formula (1-13) seems to be more convenient than (1.7), because it enables one to parametrize 
singular self-adjoint boundary conditions (at the endpoint b) by means of a self-adjoint 
boundary parameter B. 

Next assume that </?(•, A) and are [i?o, H]- valued operator solutions of equation 

(1.4) satisfying the initial conditions 

f>{aA) = {^"q^ (e [H^,Hq®H]), V'(a,A) = (l^^) [Ho,H^®H]). 

We show that, for each Nevanlinna boundary parameter r = {(Co(A), Ci(A))}, there exists 
a unique operator function mr{X){& [Ho\) such that the operator solution 

Vr{t, A) := ifi{t, A)mi-(A) + ip{t, A), A e C \ M, 

of Eq. (1.4) has the following property: for every ho G Ho the function y = Vr{t,X)hQ 
belongs to L\{I) and satisfies the boundary conditions 

i{y{a) - fby) = PqK, Co(A)Fo6y + Ci{X)Vuy = 0. 

We call mr(-) the m-function corresponding to the boundary problem (1.9)-(1.11). It turns 
out that mr(-) is a Nevanlinna operator function satisfying the inequality 

(ImA)-^ •Imm^(A) > [ v;{t,X)A{t)vr{t,X)dt, A e C \ M. 



Moreover, in the case of the Hamiltonian system the m-function of the "canonical" boundary 
problem (1.9), (1.13) coincides with the Titchmarsh - Weyl coefficient Mtw{') in the sense 
of [20, 30, 27]. Note also that a concept of the Titchmarsh - Weyl function for the general 
system (1.2) with separated A-depending boundary conditions was proposed in [27]. This 
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function is no longer a Nevanlinna function, that does not allow one to define the spectral 
function of the corresponding boundary value problem (cf. (1.16) below). 

In the final part of the paper we study eigenfunction expansions of the boundary value 
problems for symmetric systems. Namely, let r = {{Cq{-), Ci(-))} be a boundary parameter 
and let Fr{-) be the spectral function of T generated by the boundary value problem (1.9)- 

(1.11) . A nondecreasing left-continuous operator-valued function Si-(-) : M — )• [Hq] is called 
a spectral function of this problem if, for each function / e L\{I) with compact support, 
the Fourier transform 

(1.14) f{s)= J^^*{t,s)A{t)f{t)dt 
satisfies 

(1.15) ((i^r(/3)-F,(a))/,/)^^(j)= / {d^r{s)f{s)J{s)) 

for any compact interval [a, /3) C IR.. We show that for each boundary parameter r there 
exists unique spectral function S^(-) and it is recovered from the m- function m^(-) by means 
of the Stieltjes inversion formula 

1 T"* 

(1.16) St(s) = Hm Hm — / Immx(cr -|- ze) rfa. 

<5^+0£->+0 7r J _^ 

Below (within this section) we assume for simplicity that T is a (not necessarily densely 
defined) operator, i.e., mulT = {0}. 

It follows from (1.15) that, the mapping V f = f, originally defined by (1.14) for functions 
with compact supports, admits a continuous extension to a contractive map V : L\{Z) — >• 
L^(Sr; ^^o) (for the strict definition of the Hilbert space L^(S^; Hq) see [13, 24, 35] and also 
Section 6.2). In the following theorem we characterize the most interesting case when the 
mapping V is isometric. 

Theorem 1.1. For each boundary parameter t the following statements are equivalent: 

(i) The Fourier transform V is an isometry from L\{I) to L^(ST-;i?o) or, equivalently, 
the Parseval equality WTWl^ce^^Ho) = ll/llLi(i) holds for every f G L\{T). 

(ii) The exit space self- adjoint extension T"^ (in Sj) is the operator, that is mulT"^ = {0}. 
If (i) (hence (ii) ) is valid, then: 

(1) For each f e L\{I) the inverse Fourier transform is given by 

f{t)= [ ^{t,s)di:r{s)f{s) 

Jr 

where the integral is understood in an appropriate sense. 

(2) There exists a unitary extension U of the operator V that maps 9j onto L'^iY,^; Hq) 
and such that the operator T"^ is unitarily equivalent to the multiplication operator A on 
L^(Tit; Hq), T'^ = U*AU. Hence, the operators T'^ and K have the same spectral properties; 
for instance, the multiplicity of spectrum of T'^ does not exceed dim Hq (= dim H + dim H) . 

It follows from Theorem 1.1 that V is a unitary operator from L\{I) onto L'^{Y,r', Hq) 
if and only if r = {cos sin B} is a selfadjoint operator pair and the self-adjoint extension 

(1.12) of T is the operator. Observe also that the statements (i) and (ii) hold for any 
boundary parameter r if and only if T is a densely defined operator. 
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Next, we show that ah spectral functions S^(-) can be parametrized immediately in terms 
of the boundary parameter r. More precisely the following theorem holds. 

Theorem 1.2. There exists a Nevanlinna operator function 

(1.17) M(A) = {^^^^^ JJj(A)^ :Ho®Ub^Ho®Ub, A e C \ K, 

such that for each Nevanlinna boundary parameter r = {(Co(')j C'i(-))} the corresponding 
m-function mr{-) is given by 

(1.18) mr{X) = mo(A) + M2(A)(Co(A) - Ci(A)M4(A))-iCi(A)M3(A), A e C \ M. 

Thus, formula, (1.18) together with the Stieltjes inversion formula (1.16) defines (unique) 
spectral function Y,t{') of the boundary problem (1.9)-(1.11). Moreover, the Fourier trans- 
form V is an isometry if and only if the following two conditions are fulfilled: 

(1.19) lim i(Co(«y) - C\{iy)Miiiy))-'C\iiy) = 0, 

(1.20) lim ^M4iy){Co{iy) - Ci{iy)Mi{iy))-^Co{iy) = 0. 

Note that a description of spectral functions for various classes of boundary problems in 
the form close to (1.18), (1.16) can be found in [14, 16, 19, 23, 26, 40]. 

The above results are obtained in the framework of the new approach to the extension 
theory of symmetric operators developed during three last decades (see [7. 9, 10, 17, 34, 36, 
37, 39] and references therein) . This approach is based on concepts of boundary triplets and 
the corresponding Weyl functions. To apply this method to boundary value problems for 
system (1.2) we construct an appropriate boundary triplet for the relation Tmax (see Propo- 
sition 3.6). Moreover, in Proposition 4.4 and Corollary 4.5 we express the corresponding 
Weyl function M(-) in the sense of [9, 34, 39] in terms of the boundary values of respective 
matrix solutions of (1.4). It is worth to mention that the operator- valued function (1.17) 
coincides with the Weyl function M(-) computed in Corollary 4.5. Note also that conditions 
(1.19), (1.20) are implied by general result on 11- admissibility from [7, 8]. 

We complete the paper by explicit example illustrating the main results. 

Some results of the paper have been published as a preprint [2] . 

2. Preliminaries 

2.1. Notations. The following notations will be used throughout the paper: ij, V. denote 

Hilbert spaces; \Hi^T-L2\ is the set of all bounded linear operators defined on the Hilbert 
space "Hi with values in the Hilbert space 'H2] ["H] := ['H,'H]; A |" £ is the restriction of an 
operator A onto the linear manifold C; Pc is the orthogonal projector in Sj onto the subspace 
C d fy, C+ (C_) is the upper (lower) half-plane of the complex plane. 

Recall that a closed linear relation from "Ho to "Hi is a closed linear subspace in "Ho ® "Hi. 
The set of all closed linear relations from 'Ho to 'Hi (in %) will be denoted by C('Ho,'Hi) 
{C{'H)). A closed linear operator T from Ho to Hi is identified with its graph grT e 
C{Ho,Hi). ^ 

For a linear relation T G C{Ho,Hi) we denote by domT, ranT, kerT and mulT the 
domain, range, kernel and the multivalued part of T respectively. Recall also that the inverse 
and adjoint linear relations of T are the relations T~^ e C{Hi,Ho) and T* e C{Hi,Ho) 
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defined by 

T-i = {{hi, ho} e Til® no: {ho, hi} G T} 

(2.1) T* = {{fci, fco} e Hi e Ho : {ko, ho) - {kiM) = 0, {ho, hi} e T}. 

In the case T G C{Ho,'Hi) we write G p{T) if kerT = {0} and ranT = Hi, or 
equivalently if G [Hi, Ho]; G p(T) if kerT = {0} and ranT is a closed subspace in 
Hi. For a linear relation T G C(H) we denote by p{T) := {A G C : G p(T - A)} and 
p(T) = {A G C : G p(T — A)} the resolvent set and the set of regular type points of T 
respectively. 

A linear relation T G C(H) is called symmetric (self-adjoint) if T C T* (resp. T = T*). 
For each T = T* G C(H) the following decompositions hold 

(2.2) H = H'®mulT, T T' © lii^ii T, 

where mulT = {0} © mulT and T' is the self-adjoint operator in H' (the operator part of 

Let T = T* G C(H), let B be the Borel cr-algebra of M and let E'{-) : B ^ [W] be 
the orthogonal spectral measure of T'. Then the spectral measure E{-) of T is defined as 
E{5) = E'{S)Ph', SgB. 

Recall also the following definition. 

Definition 2.1. A holomorphic operator function $(•) : C\M — )• [H] is called a Nevanlinna 

function if Im A • Im$(A) > and $*(A) = $(A), A G C \ R. 

2.2. Holomorphic operator pairs. Let A be an open set in C, let /C,Ho,Hi be Hilbert 
spaces and let 

(Co(A), Ci(A)) iHoffiHi ^/C, AG A, 
be a pair of holomorphic operator functions Cj(-) : A — >• [Hj,/C], j G {0,1} (in short a 
holomorphic pair). Two such pairs Cj{-) : A — >• [Hj, /C] and Cj(-) : A — >• [Hj,/C'] are said 
to be equivalent if there exists a holomorphic isomorphism ip(-) : A — ^ [/C,/C'] such that 
Cj(A) = (p(A)Cj(A), A G A, j G {0, 1}. Clearly, the set of all holomorphic pairs splits into 
disjoint equivalence classes; moreover, the equality 

(2.3) t(A) = {(Co(A),Ci(A));/C} := {{ho,hi} G Ho®Hi : Co(A)/io + Ci(A)/ii = 0} 

allows us to identify such a class with the C(Ho,Hi)-valued function t(A), A G A. 

In what follows, unless otherwise stated. Ho is a Hilbert space. Hi is a subspace in Ho, 
H2 := Ho Hi and Pj is the orthoprojector in Ho onto Hj, j G {1,2}. 

With each linear relation 6 G C(Ho,Hi) we associate the x-adjoint linear relation 6^ G 
C(Ho,Hi) given by 

={{feo,fci}€Ho®Hi :{ki,ho)-{ko,hi)+i{P2ko,P2ho)=0, {ho,hi}€e}. 

It follows from (2.1) that in the case Ho = Hi =: H one has 9^ = 6* . 

Next assume that 
.24^ T+(A) = {(Co(A),Ci(A));Ho}, A G C+; 

^ ^ ^ r_(A)={(Z)o(A),Di(A));Hi}, A G C_ 

arc equivalence classes of the holomorphic pairs 

(2.5) (Co(A),Ci(A)):HoeHi^Ho, A G C+ 

(2.6) (I?o(A),T'i(A)) :Ho©Hi ^Hi, A G C_. 
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Assume also that 

Co(A) = (Coi(A),Co2(A)) : 7^1 ® 7^2 ^ Ho 
DoiX) = poi(A), Do2iX)) ■.Hi(BH2^Hi 
are the block representations of Co (A) and -Do (A). 

Deflnition 2.2. A coUection r = {t+,t_} of two holomorphic pairs (2.4) (more precisely, 
of the equivalence classes of the corresponding pairs) belongs to the class R{%q,'Hi) if it 
satisfies the following relations: 



(2.7) 2 Im(Ci (A)Co*i (A)) + Co2(A)Co*2(A) > 0, 

(2.8) 2Im(Di(A)D*i(A)) +£>02(A)7?S2(A) < 0, 

(2.9) Ci(A)D*i(A) - Coi{\)Dl(\) + iCo2(A)D*2(A) = 0, A € C+ 

(2.10) Oep(Co(A)-iCi(A)Pi), AeC+; € p(Doi(A) + i£'i(A)), A e C_. 



A collection r = {T+,r_} G i?('Ho,'Hi) belongs to the class E?{%q,%\) if for some (and 
hence for any) A G C+ one has 

2Im(C'i(A)Co*i(A)) + Co2(A)Co*2(A) = and € p(Coi(A) + zCi(A)). 

The following proposition is immediate from Definition 2.2 and the results of [38]. 

Proposition 2.3. (1) If t = {T+,r_} e R{Ho,Hi), then (-r±(A))^ = -r:p(A), A G C^, 
and the following equality holds 

(2.11) T^(A) = {{-/ii -iP2/io,-Pi/io} : {hi,ho} e (r±(A))*}. 

(2) T/ie set R°{Ho,y-i) is not empty if and only if dimHo = dim 7/1. This implies that 
in the case dim "Hi < 00 the set R'^iHo^Hi) is not empty if and only if 'Ho = T-Li =:T-L. 

(3) Each collection t = {r+,r_} e RP{%o,'Hi) can he represented as a constant 

(2.12) T±{X) = {{Co,C^);Ho} = e{&C{Uo,U^)), A e C±, 
where C, e [Uj,UQ], j € {0, 1}, and (-6*)^ = -0. 

Moreover, one can easily prove the following proposition. 

Proposition 2.4. //dim "Ho < 00, then a collection t = {r+,T_} of two holomorphic pairs 
(2.4) belongs to the class R{'Hq,'Hi) if and, only if (2.7)-(2.9) holds and 

(2.13) ran(Co(A),Ci(A)) =7^0, A e C+; ran (L»o(A), A (A)) = Hi, A G C_. 

Remark 2.5. If Hi = Hq =: then the class R{T-L) := R{'H,'H) coincides with the well- 
known class of Nevanlinna functions t(-) with values in C(H) (see, for instance, [7]). In this 
case the collection (2.4) turns into the Nevanlinna pair 

(2.14) r(A) = {(Co(A),Ci(A));H}, AeC\R, 

with Co(A), Ci(A) € [H]. In view of (2.7)-(2.10) such a pair is characterized by the relations 
(cf. [7, Definition 2.2]) 

(2.15) ImA-Im(Ci(A)Co*(A)) > 0, Ci(A)Co*(A) - Co(A)Ci*(A) = 0, A e C \M, 

(2.16) Oep(Co(A)-iCi(A)), AeC+; e p(Co(A) + iCi(A)), A e C_. 
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Moreover, the function t(-) belongs to the class R^{'H) R^{'H,'H) if and only if it admits 
the representation in the form of the constant (cf. (2.12)) 

(2.17) r(A) = {(Co,Ci);H} = 0(e CCH)), AeC\M 

with the operators Cj G [H] such that Im(CiCo) = and G p{Co ± iCi) (this means 
that 6 = 6*). Observe also that according to [45] each r € BP{H) admits the normalized 
representation (2.17) with 

(2.18) Co = cosS, Ci = sinS, B = B* € [K]. 

Assume now that n := dim'H < oo, e = {ej}" is an orthonormal basis in H, t(A) = 
{(Co(A), Ci(A));'H} is a pair of holomorphic operator-functions Cj(-) : C \ M — >■ [H] and 
Ci{\) = {ckj,i{^))kj=i is the matrix representations of the operator Cz(A), I € {0, 1}, in the 

basis e. Then by Proposition 2.4 r belongs to the class R{H) if and only if the matrices 
Co(A) and Ci(A) satisfy (2.15) and the following equality: 

rank(Co(A),Ci(A)) =n, AeC\M. 

Moreover, the operator pair 6 ~ {(Co,Ci);'H} belongs to the class R^CH) if and only if 
Im(CiCo) = and rank (Co, Ci) = n (here C; = {ckj,i)kj=i is the matrix representation 
of the operator C I € {0, 1}, in the basis e). Note that such a "matrix" definition of the 
classes J?('H) and R^{H) in the case dim'H < oo can be found, e.g. in [12, 29] 

2.3. Boundary triplets and Weyl functions. Here we recall definitions of boundary 

triplets, the corresponding Weyl functions, and 7-fields following [9, 10, 34, 39]. 

Let A be a closed symmetric linear relation in the Hilbert space ^ , let {A) = ker {A* — 
A) (A e p{A)) be a defect subspace of A, let ^x{A) = {{/,A/} : / e ^\{A)} and let 
n±{A) := dim*JlA(^) < 00, A G C±, be deficiency indices of A. Denote by ExIa the set of 
all proper extensions of A, i.e., the set of all relations A e C(^) such that Ad Ad A* . 

Next assume that 'Ho is a Hilbert space, 'Hi is a subspace in "Hq and 'H2 '■= HoQ'Hi, so 
that Ho =Hi ® %2- Denote by Pj the orthoprojector in "Ho onto 'Hj, j G {1, 2}. 

Definition 2.6. A collection H = {Ho ?^i,ro,ri}, where Tj : A* Hj, j G {0, 1}, are 

linear mappings, is called a boundary triplet for A* , if the mapping F : / — > {ro/,ri/}, / G 
A* , from A* into Ho ® Hi is surjcctive and the following Green's identity 

(2.19) (/', g) - if, g') = (rj, Tog)no " i^ol rig)Ho + iiP^^ol P2Tog)H, 
holds for all / = {/, /'}, g = {g, g'} G A*. 

Proposition 2.7. Let IT = {Ho ® "Hi, Fg, Fi} he a boundary triplet for A* . Then: 

(1) dim-Hi = n_(A) < n+(A) = dim-Ho• 
(2) ker To nkerFi = A and Tj is a hounded operator from A* into Hj, j G {0, 1}. 
(3) The equality 

(2.20) Ao := ker Fo = {/ G A* : Fo/ = 0} 

defines the maximal symmetric extension Ao G Ext a such that C+ C p{Ao). 

Proposition 2.8. [39] LetH = {Ho®Hi,To,Ti} he a boundary triplet for A* . Denote also 
byiTi the orthoprojector in Sj®Sj onto S)(B{0}. Then the operators To \ yt\{A), A G C+, and 
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PiFq |~OTz(A), z€ C_, isomorphically map Vl\{A) ontoHo and^zi^) ontoHi respectively. 

Therefore the equalities 

.2 21) = ^1(^0 r ^x{A))-\ A e C+, 

(2.22) M+(A)/io = ri{7+(A)/io,A7+(A)/io}, hoeno, A G C+ 

(2.23) M_{z)hi = (Fi + iP2To){j-iz)hi,z^-{z)hi}, hi eHi, z€ C_ 

correctly define the operator functions 7+(-) : C+ — >■ [HojSj], 7-(-) : C_ ^ [Hijio] and 
M+(-) : C+ [HojHi], M_(-) : C_ — > ["Hi, "Ho],- w^/'-^cfe are holomorphic on their domains. 

Moreover, the equality A'/^(A) ~ M_(A), A G C-, is valid. 

It follows from (2.21) that for each Hq G Hq and hi E Hi the following equalities hold 

(2.24) To{7+{X)ho,X7+Who} = ho, PiTa{"f-{z)hi, z^f^{z)hi} = hi. 

Definition 2.9. [39] The operator functions 7±(-) and M±{-) defined in Proposition 2.8 
are called the 7-fields and the Weyl functions, respectively, corresponding to the boundary 
triplet n. 

Proposition 2.10. Let H — {Ho ® HijTqjTi} be a boundary triplet for A* and let 7±(-) 
and M±{-) be the corresponding ^-fields and Weyl functions respectively. Moreover, let the 
spaces Ho and Hi be decomposed as 

Hi = H ® Hi, Ho = H® Hq 

(so that Ho =Hi®H2) and let 

To = (Po, f 0)^ ■.A*^H®Ho, Ti = (Pi, f 1)^ :A*^H®Hi 

be the block representations of the operators To and Ti . Then: 

(1) The equality 

A={feA*:fof = tof = tif = 0} 
defines a closed symmetric extension A e Ext a and the adjoint relation A* of A is 

A* = {feA* ■.Tof = Q}. 

If in addition n±{A) < 00, then the deficiency indices of A are n±{A) = n±{A) — dim'H. 

(2) The collection 11 = {Hq (BHi,To \ A* ,Ti \ A*} is a boundary triplet for A* . 

(3) The ^-fields 7±(-) and the Weyl functions M±{-) corresponding to 11 are given by 

7+(A) = 7+(A) \ Ho, M+iX) = P^^M+(A) \ Ho, A e C+ 
7-(A)=7-(A) r^i, M_(A)=P^„M_(A) r^i, A e C_. 

We omit the proof of Proposition 2.10, since it is similar to that of Proposition 4.1 in [7] 
(see also remark 2.11 below). 

Remark 2.11. If Ho = Hi := H, then the boundary triplet in the sense of Definition 2.6 

turns into the boundary triplet 11 = {H,To,Ti} for A* in the sense of [17, 34]. In this case 
n+(A) = n-{A) = dim'H, Ao{= kerFo) is a self-adjoint extension of A and according to 
[9, 34, 10] the relations 

(0 2^) j{X) = ni{To \ mx{A))-\ 

^' ' ri{7(A)/i,A7(A)/i} = M(A)/i, h€H, X e p{Ao) 
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define the 7-field 7(-) : p{Aq) [V.,S)] and the Weyl function M(-) : p{Ao) [%] cor- 
responding to the triplet 11. It follows from (2.25) that 7(-) and M(-) are associated 
with the operator functions 7±(-) and M±(-) from Definition 2.9 via 7(A) = 7±(A) and 
M(A) = M±(X), A € C±. Moreover, for such a triplet the identity 

(2.26) M(/x) -M*(A) = (m- A)7*(A)7(At), A G C \ M. 

holds, which implies that Af(-) is a Ncvanlinna operator function. Observe also that for the 
triplet n = {H, To, Fi} all the results in this subsection were obtained in [9, 34, 10, 7]. 

In what follows a boundary triplet 11 = {H.Tq^Vi^ in the sense of [17, 34] will be 
sometimes called an ordinary boundary triplet for A*. 

2.4. Generalized resolvents and spectral functions. Let 5o be a subspace in a Hilbert 

space Jo, let A — A* ^ C(Jo) and let E{-) be the spectral measure of A. 

Definition 2.12. The relation A is called .Q-minimal if it satisfies at least one of the following 
equivalent conditions: 

(1) span{^, (I - A)-ii3 : A G C \ R} = ^; 

(2) there is not a nontrivial subspace Sj' C 9 such that £■([«, /3))^' C ij' for each 
bounded interval [a, /?) C K. 

Definition 2.13. The relations Tj G C{^j), j G {1,2}, are said to be unitarily equivalent 
(by means of a unitary operator U G [i5i,i52]) if ^2 = UTi with U = U®U e [Sji,Sjl]. 

Proposition 2.14. Let Sjj be a subspace in a Hilbert space and let Aj — A* £ C{S)j) be 
a S^j-minimal relation, j G {1,2}. Assume also that V G [i5i,i52] is a unitary operator such 
that 

P^Mi-\)-^ tiOi =y*(Pr,,(l2-A)-i \^2)V. 

Then there exists a unitary operator U G [i5i,i52] such that U \ S)\ = V and the relations 
Ai and A2 are unitarily equivalent by means of U. 

In the case iji = S^2 ='■ and V = Isj the proof of this proposition can be found in [31]. 

In general case the proof is similar. 

Recall further the following definition. 

Definition 2.15. Let A be a symmetric relation in a Hilbert space S). The operator func- 
tions R{-) : C \ M — )• [Sj] and F(-) : K — >■ [ij] are called the generalized resolvent and the 
spectral function of A respectively if there exist a Hilbert space D and a self-adjoint 
relation A G C{Sj) such that A C A and the following equalities hold: 

(2.27) R{X) ^ Ps,{A- X)-^ \ Sj, AgC\M 

(2.28) F{t) ^ PsjE{{-oo,t)) \ Sj, teR 

(in formula (2.28) E{-) is the spectral measure of A). 

The relation A in (2.27) is called an exit space extension of A. 

It follows from (2.27) and (2.28) that the generalized resolvent R{-) and the spectral 
function F{-) generated by the same extension ^4 of ^ are connected by 

(2.29) i?(A)= / AGR. 
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Moreover, (2.28) yields 

(2.30) F{oo){:= s - ^ Urn^ F{t)) = P^P~^^ \ Sj, 

where Sjo = SjQ mul A. 

According to [31] each gcncrahzed resolvent of A is generated by some i^-minimal exit 
space extension A of A. Moreover, if the ij-minimal exit space extensions Ai S C{Sji) and 
A2 € C{Sj2) of A induce the same generalized resolvent R{X), then in view of Proposition 
2.14 there exists a unitary operator V € [Joi QSj,Sj2QSj] such that Ai and A2 are unitarily 
equivalent by means of C/ = I^j S)V'. By using this fact we suppose in the following that 
the exit space extension A in (2.27) is j^-minimal, so that A is defined by R{-) uniquely up 
to the unitary equivalence. 

Definition 2.16. The generalized resolvent (2.27) and the spectral function (2.28) are called 

canonical if 5o = f), i.e., if i?(A) = {A — X)~^, A € C \ M, is the resolvent of the extension 
A = A* e C{^) of A and F{t) = E{{~-(yD, t)), te R, is the spectral function of A. 

Clearly, canonical resolvents and spectral functions exist if and only if n+{A) = n_(A). 
A description of all generalized resolvents of A in terms of boundary triplets for A* is 

given in the following theorem (see [6, 34] for the case Ti+{A) = n-{A) and [39] for the case 
of arbitrary deficiency indices n±{A)). 

Theorem 2.17. LetIL = {Ho®'Hi,TaSi} he a boundary triplet for A* . //T = {T+,T_}e 
R{T-Lo,'Hi) is a collection of holomorphic pairs (2.4), then for every g £ Sj and A e C \ R 
the abstract boundary value problem 

(2.31) {f,xf + g}GA* 

(2.32) Co(A)ro{/,A/ + 5}-Ci(A)ri{/,A/ + g} = 0, A G C+ 

(2.33) Do{X)ro{f,Xf + 9}-DiiX)T,{f,\f + g} = 0, A G C_ 

has a unique solution f = f{g,X) and the equality R{X)g := /(g. A) defines a generalized 
resolvent R(X) = Rt{X) of the relation A. Conversely, for each generalized resolvent R{X) 
of A there exists a unique r e R{'Ho,Hi) such that R{X) = Rt{X). Moreover, Rr{X) is a 
canonical resolvent if and only if t G i?°('Ho,'Hi). 

3. Boundary triplets for symmetric systems 

3.1. Notations. Let I = [a,b) (—00 < a < b < 00) be an interval of the real line (the 

symbol ) means that the endpoint 6 < 00 might be either included to I or not). Further, let 
H be a finite-dimensional Hilbert space, let AC(I;B[) be the set of functions /(•) : I — >• HI 
which are absolutely continuous on each segment [a,^] C X and let ACil) := ^C(I;C). 
Denote also by C\^^{I: [H]) the set of Borel operator- valued functions F{-) defined almost 
everywhere on T with values in [H] and such that / ||-F(t)|| dt < 00 for each /? € I. 

Next assume that A(-) € Cj^^il; [H]) is an operator function such that A(t) > a.e. on 
I. Denote by >C^(X) the hnear space of all Borel-measurable vector-functions /(•) : X — > H 
satisfying 

j{A{t)f{t),f{t))ndt = j ||A5(f)/(f)||2rft<oo. 
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Moreover, for a given finite-dimensional Hilbert space K. denote by /!^[/C,]HI] the set of all 
Borel operator-functions F{-) : X — > [/C,]HI] such that F{t)h S 'C^(X) for each h & fC. It is 

clear that the latter condition is equivalent to / ||A2(t)F(t)|p dt < oo. 

I 

It is known [24, 13, 35] that is a semi-Hilbert space with the semi-definite inner 

product (•, •)a and the semi-norm || • ||a given by 

(3.1) {f,g)^ = jm)f{t),g{t))ndt, ||/||A = ((/,/)A)^ /,5e£i(I). 

The semi-Hilbert space C\{L) gives rise to the quotient Hilbert space L\{I) = C\{I)/{f G 
^\^) '■ I I/I I A = 0}. The inner product and the norm in L\il) are defined by 

(/,?) = (/,<7)a, ||/|| = (/,/)^ = ||/||a, J,g&Ll{Il 

respectively, where f G f {g &g) is any representative of the class / (resp. g). 

In the sequel we systematically use the quotient map tt from C\il) onto L\il.) given 
by tt/ = 1(5 f), / G Clil). Moreover, we let ^ = tt 8 tt : {Cl{I))^ (^'A(^))^ so that 
n.f,9} = {f,9}, f,g&C\{l). 

3.2. Symmetric systems. In this subsection wc provide some known results on symmetric 

systems of differential equations following [15, 25, 28, 33, 44]. 

Let as above I = [a, b) (— oo < a <h < oo) be an interval and let H be a Hilbert space 
with n := dimH < oo. Moreover, let _B(-), A(-) e Cl^^{I; [H]) be operator functions such 
that B{t) — B*{t) and A(t) > a.e. on I and let J G [H] be a signature operator ( this 
means that J* = = — J). 

A first-order symmetric system on an interval I (with the regular endpoint a) is a system 
of differential equations of the form 

(3.2) Jy'{t)-B{t)y{t) = A{t)f{t), tel, 

where /(•) S C\{X). Together with (3.2) we consider also the homogeneous system 

(3.3) Jy'{t) - B{t)y{t) = XA{t)y{t), tGl, A e C. 

A function y G AC{X;M) is a solution of (3.2) (resp. (3.3)) if the equality (3.2) (resp. (3.3) 
holds a.e. on I. Moreover, a function Y{-,X) : I — >■ [/C,IHI] is an operator solution of the 
equation (3.3) if y{t) = Y{t, X)h is a (vector) solution of this equation for each h £ IC (here 
/C is a Hilbert space with dim/C < oo). 

In what follows we always assume that system (3.2) is definite in the sense of the following 
definition. 

Definition 3.1. [15, 28] The symmetric system (3.2) is called definite if for each A G C and 
each solution y of (3.3) the equality A{t)y{t) — (a.e. on X) implies y{t) =0, t E X. 

As it is known [44, 25, 33] symmetric system (3.2) gives rise to the maximal linear relations 
T^ax and T^ax in and L\{I), respectively. They are given by 

4^ Tm.. = {{y, /} e (£i(J))2 : y G AC{X; H) and 

^ ' Jy'{t) - B{t)y{t) = A{t)f{t) a.e. on 1} 

and Tmax = TrT^ax- Moreover the Lagrange's identity 

(3.5) {f,z)A- {y,g)A = [y,z]b- {Jy{a),z{a)), {y, f}, {z,g} eTma^. 
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holds with 

(3.6) [y, z]b := lim{Jy{t), z{t)), y,ze dom7;„ax. 

Formula (3.6) defines the boundary bilinear form [■,■]& on domTInax: which plays a crucial 
role in our considerations. By using this form we define the minimal relations Tmin in 
and Tmin in L\{I) via 

T^in = {{y, /} e T^ax : y{a) = and [y, z\b = for each z G domTl^nax}- 
and Tmin = T^Tmin- According to [44, 33] Tmin is a closed symmetric linear relation in L\{I) 

and T* — T 
d^^^ -'■min — -'■max* 

Remark 3.2. It is known (sec e.g. [33]) that the maximal relation Tmax induced by the 
definite symmetric system (3.2) possesses the following regularity property: for each {y, /} G 
Tmax there exists unique function y G ACiX; H) n C\{X) such that y Gy and {y, /} G T^ax 
for each f G f- Below we associate such a function y e AC{I;'S} n jC\{X) with each pair 

{y, 1} e Tmax. 

For any A G C denote by Afx the linear space of solutions of the homogeneous system 
(3.3) belonging to £\{X). Definition (3.4) of T^ax implies 

Afx = kcr (Tmax -X) = {y€ Clil) : {y, Xy} G Tmax}, A G C, 

and hence J\fx C domT^ax- 
As usual, denote by 

n±(Tmin) = dimOTA(7;nin), A G C±, 

the deficiency indices of Tnin- Since the system (3.2) is definite, ttJVx = OTA(Tmin) and 
ker (tt [ J\fx) = {0}, A G C. This implies that dimAA^ = n±(Tmin), A G C±. 
The following lemma will be useful in the sequel. 

Lemma 3.3. (1) IfY(-,X) G £^[/C,H] is an operator solution of Eq. (3.3), then the relation 

(3.7) ICb h^ {Y{X)h){t) ^Y{t,X)heAfx. 

defines the linear mapping Y{X) : K — )• Nx and, conversely, for each such a mapping Y[X) 
there exists unique operator-valued solution Y{-,X) G £^[/C, H] of equation (3.3) such that 

(3.7) holds. 

(2) Let y(-, A) G £^[/C,H[be an operator solution of Eq. (3.3) and let F(A) = 7ry(A)(G 
[K,L\{I)\). Then for each / G L\{I) 

(3.8) F*{X)f = I^Y*{t,X)A{t)f{t)dt, fef. 

The first statement of this lemma is obvious, while the second one can be proved in the 
same way as formula (3.70) in [41] (see also formula (2.40) in [33]). 
Let J G [M] be the signature operator in (3.2) and let 

i/+ = dim ker {iJ — I) and V- = dim ker {iJ + 7) . 

In what follows we suppose that 

(3.9) V := V. - v+> 0. 

In this case one can assume without loss of generality that the following statements hold: 
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(i) the Hilbert space H is of the form 

(3.10) m = H®H®H, 

where H and H are finite dimensional Hilbert spaces with 

(3.11) dimH = v+, dimH = V] 

(ii) the operator J is of the form (1-3). 
Introducing the Hilbert space 

(3.12) Ho = H®H 
one can represent the equality (3.10) as 

(3.13) m={H®H)®H = Ho®H. 

Let Vb+ and vi,- be inertia indices of the skew-Hermitian bilinear form (3.6). Then 
yi)± < DO and the following equalities hold [4, 42] 

(3.14) n+(Tmin) = + Z^6+, n_(Tmin) = I^- + fb-. 

This yields the equivalence 

(3.15) n+(Tmin) = n-(rmin) <S=^ I? = f 6+ - f 6_ . 

Next assume that 

(3.16) [/=f"^ ""A-.H^H^H^H^H 

is the operator satisfying the relations 

(3.17) ran;7 = ^®F 

(3.18) iu2U2 — M1U3 + U3?i* = ilg, iu^U2 — U4U3 + uqu{ = 

(3.19) iu^u^ + U6M4 — UiUg — 

One can prove that the operator (3.16) admits an extension to the J- unitary operator 





fur 


Us 




(3.20) 


U=\ui 




Us 1 




\U4 


U5 





i.e. the operator satisfying U*JU = J. The operator (3.20) induces the linear mapping 
r„ : AC{I; e) H given by 

(3.21) Tay = Uy{a), y€AC{I;m). 

In accordance with the decomposition (3.10) admits the block representation 

(3.22) r, = (roa, r„ Ti,) ^ : ACil; M) ^ H (B H (B H. 
If a function y € AC{X; H) is decomposed as 

y{t) = {yo{t), y{t), yi{t)]{& h®h®h), tei, 
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then the mappings Vj^ : AC{I;m) H, j G {0, 1}, and f„ : AC(2:;H) ^- ^ in (3.22) can 
be represented as 

(3.23) ToaU = U7yo{a) +usyia) + ugyi{a), y e AC {I;B.) 

(3.24) Tay = Miyo(a) + U2y{a) + U3yi{a), Tiay = u^yoia) + ur,y{a) + U62/i(a)- 

This imphes that and Tia are determined by the operator U , while Foa is determined by 
the extension U . 

Let A e C and /C be a finite-dimensional Hilbcrt space. By using the operator (3.20) wc 
associate with each operator solution Y{-,\) : X — )• [/C,!!] of equation (3.3) the operator 
y„(A)e[/C,e] given by 

(3.25) Y^{\) = UY{a,\). 

If in addition Y{-, A) e £^[/C, H], then the operator (3.25) admits the representation 

(3.26) y„(A) = r„y(A), 

where ^(A) is defined in Lemma 3.3. 

In what follows we associate with each operator U (see (3.16)) the operator solution 
ip{-,\) = (yj(7(-, A)(g [i?o,H]), A G C, of Eq. (3.3) with the initial data 

( ul iul\ 

(3.27) ^u{a,X)= \-iul ul : H (B H ^ H ® H ® H . 

\~'^% Ho H 

One can easily verify that for each J-unitary extension U of U the following equality holds 

(3.28) maW{= U^uia, A)) = (^^JJ"^ :Ho^Ho®H. 

The particular case of the operator U and its J-unitary extension U is (cf. [21]) 

( n \ ~ / BO— cos 

D n - d)' U=\ Iff 
cosB smBI \ u n ■ u 

' \cosB smi? 

where B = B* £ [H]. For such U the solution (pu{-, A) is defined by the initial data 

/ sinS \ 
ipu{a,X)=\ Iq\ : H®H ^ H®H®H. 
\-cosB / 

3.3. Decomposing boundary triplets. We start with the following lemma. 

Lemma 3.4. //n_(Tinin) < '^-i-(7min); then there exist a finite dimensional Hilbert space 
Tib, 0- subspace Tib C Hb and a surjective linear mapping 

(3.29) Tb = Ffe : domTmax ^Hb(S)H®Hb 

such that for all y,z G domTmax the following identity is valid 

(3 30) " ^i''^)w6 ~ ^^^by, Tobz)f^^ + 

' +i{PHi'^oby,Pni^obz)^^ + i{fby,fbz)fj 
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Moreover, for each such a mapping one has 

(3.31) dimHb = Vb-, dim'Hb = Vb+ — v 
and the following equivalence holds 

(3.32) n+(Tinin) = "-(Tinin) ■^=^ Tib = Tib- 

Therefore in the case of equal deficiency indices n+(Tniin) = n_(Ti„i„) the identity (3.30) 
takes the form 

[y,z]b = {Toby,Tibz)n„ - iTiby,Tobz)nt + ii^'by,^'bz)H 

Proof. In view of (3.14) and (3.9) one has i'b+ — i^b- > J^- Therefore by [42, Lemma 5.1] 
there exist Hilbert spaces T-Lb and Hb and a surjective linear mapping 

(3.33) Fft = (F^,;,, r;, TuV •■ domTma^ ^Hb(BUb(BHb 
such that 

[y, z]b = (Fob?;, Fi6z)«, - (Fiby, Fg^z)-?^^ + i(F(,y, F^z)^^, y,z G domTmax- 
Moreover, for such a mapping F;, one has 

(3.34) dim'Hb = i'b-, dim'Hf, = — 

which in view of (3.11) yields dimHb > dimH. Therefore without loss of generality one 
may assume that H cHb and hence 

(3.35) Hb = U'2®H 

with ='HbQH. Let 'Hb = T-Lb® '^2 (so that Tib C Tib) and let : domTmax Hb and 
Tb : domTmax H he the linear mappings given by 

Tob = Fq5 + P-Hi^T'^, = PgF'f,. 

Then (3.33) can be written in the form (3.29) and the direct calculation gives the identity 
(3.30). Moreover, 

dimHb = dim'Hb + (dim'Hb — dimH), 

which together with (3.34) and the second equality in (3.11) yields (3.31). Finally, the 
equivalence (3.32) is implied by (3.15) and (3.31). □ 

Remark 3.5. (1) Since the mapping Fb is surjective, it follows from (3.30) that Fby = for 
each function y S domT^nax such that y{t) = on some interval (/?, 6) C I. Therefore, if 
t/1,2/2 € domT^nax and yi{t) = y2{t) on some interval (^,&), then Fbyi = Tby2- 

(2) In the case of the regular system (3.2) (i.e., when X = [a, b] is a compact interval and 
both integrals Jj- \\B{t)\\ dt and jj- ||A(t)|| dt are finite) one can put in (3.29) Hb = H-b = H 
and Fby = Xby{h), y G domTmax, where Xb € [H] and X^JXb = J. 

In general case Remark 5.2 in [42] implies that the mapping (3.29) can be constructed 
with the aid of the following assertion: 

— there exist systems of functions {Oj^^ji"*' " , and {O^pYi' m domTmax such 

that the operators 

^o6y = {[^/,e?^]6}^-^ fby = T.l[y,ef\ei, T,by = {[y,6f]b}\^- 

{y e domTmax) form the surjective linear mapping Fb = (Fob, Tb, Fib)""" : domTmax — >■ 
C^b+'P ^H® €"'•- satisfying the identity (3.30) (here {e^jf is an orthonormal basis in H). 
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In the case Vb- = 0, i'b+ = y and 6xm.H = 1 one has Hb =^b = {0}. In this case one 
can put 

Tby = [y,0]be, 

where e is an ort in H and is a function in domT^ax such that [9,9]b = i- 

These assertions show that one may consider FbU as a singular boundary value of a 
function y € domT^iax (cf. [13, Ch. 13.2]). 

The following proposition is immediate from [42, Theorem 5.8] and (3.32). 

Proposition 3.6. Assume that n_(Tniin) < n+(rijiin), U is the J-unitary operator (3.20), 
is the linear mapping (3.21) with the block representation (3.22) and Ff, is the surjective 
linear mapping (3.29) satisfying the identity (3.30). Moreover, let Ho and 'Hi(c Ho) be 
finite dimensional Hilbert spaces defined by 

(3.36) Ho = Ho® Hb, Hi=Ho(B Hb 
and let Tj : Tmax — Hj, j G {0, 1}, be the operators given by 

(3.37) Toiy, /} = {-T^^y + i(f„ - fb)y, Toby}{e Ho 8 Hb), 

(3.38) ri{^, 7} = {ToaV + 5(f a + Tb)y, -ri6y}(e Ho e Hb), {y, 7} e T^a.. 

(here y G domT^ax is the function corresponding to {y,f} € Tlnax according to Remark 
3.2). Then the collection H = {Ho © Hi, To, Fi} is a boundary triplet for Tmax- 

// in addition n+(Tmin) = n-{T^in), then H turns into an ordinary boundary triplet 
n = {H,To,Ti} for Tmax, where H = Ho^® Hb and Tj : T^ax H, j G {0,1}, are the 
operators given by (3.37) and (3.38) with Hb = Hb. 

Definition 3.7. The boundary triplet 11 — {Ho ® 'Hi,Fo,ri} constructed in Proposition 
3.6 will be called a decomposing boundary triplet for Tmax- 

Proposition 3.8. Let n-{Tynin) < ^^+(^min), let U be the operator (3.16), let F^ and Fio 
be the linear mappings (3.24) and let Tb be the linear mapping (3.29). Then: 

(1) The equalities 

(3.39) T = {{y, 7} e T^ax : Ti^y = 0, f,y = fby, Toby = Tuy = 0} 

(3.40) T* = {{y, 7} e T^a. : Ti^y = 0, f^y = fby} 

define a symmetric extension T of Tmin and its adjoint T* . Moreover, the deficiency indices 
of T are n+ (T) = Vb+ — v and n_ (T) = Vb- . 

(2) The collection II = {Hb ® Hb,To,T\\ with the operators 

(3.41) to{y,f} = Toby, ti{yJ} = -Tiby, {y,f}GT*, 

is a boundary triplet for T* and the (maximal symmetric) relation Ao{= kerFo) is of the 

form 

(3.42) Ao = {{y, f} € T^ax : T^^y = 0, fay = fty, Toby = 0}. 

If in addition n+{Tjnin) = ?J-(T'mm), then n+{T) = n-{T) = Vb-, 11 = {'H6,ro,ri} is an 
ordinary boundary triplet for T* and Ao = . 

Proof. Let U be the J-unitary extension (3.20) of U, let Toa be the operator (3.23) and 
let n = {Ho © Hi,To,Ti} be the decomposing boundary triplet (3.37), (3.38) for T^ax- 
Applying to this triplet Proposition 2.10 one obtains the desired statements. □ 
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Remark 3.9. Clearly, mulT = mulT* if and only if the following condition is fulfilled: 

(CI) For each function y G domTInax the equalities 

(3.43) Flay = 0, fay = fby and A{t)y{t) = (a.e. on I) 

yield Tobj/ = Fuy = 0. 

Moreover, mulT* = (i.e., T is a densely defined operator) if and only if the following 

condition is satisfied: 

(C2) For each y e domTlnax the equalities (3.43) yield y = 0. 

4. £^-SOLUTIONS OF BOUNDARY VALUE PROBLEMS 

In what follows wc suppose that the symmetric system (3.2) satisfies the condition 
n_(Tmi„) < n+(Tnii,i). Our considerations will be also based on the following assumptions: 

(Al) U is the operator (3.16) satisfying the relations (3.17) - (3.19) and Fa and Fi^ are 

the linear mappings (3.24). 
(A2) Tib and Hbic Hh) arc finite dimensional Hilbert spaces and F;, is the surjective linear 

mapping (3.29) such that (3.30) holds. 

In addition to (A1)-(A2) we will sometimes use the following asstunption: 

(A3) U is a J-unitary extension (3.20) of U and Foo is the mapping (3.23). 

Let (A1)-(A2) be satisfied and let r = {r+,T_} e R{Hb,'Hb) be a collection of holo- 
morphic operator pairs (2.4) with Co(A) € [Hb], C\{\) € [H.b.l-Ll]^ A e C+, and Dq{\) € 
[Hb,'Hb\-i -Di(A) G [Hb], A G C_. For a given function / G 'C^(I) consider the following 
boundary value problem: 



(4.1) Jy' -B{t)y = XA{t)y + A{t)fit), t€l, 

(4.2) Fi„t/ = 0, fay = fby, AeC\M, 

(4.3) Co (A)Fo6t/ + Ci {X)Tuy = 0, A G C+ 

(4.4) £>o(A)Fo6y + £>i(A)Fi6y = 0, A G C_. 



A function y{-,-) : I x (C \ R) ^ H is called a solution of this problem if for each A G C \ M 
the function y(-,A) belongs to AC(I;EI) n£^(I) and satisfies the equation (4.1) a.e. on I 
(so that y G domT^ax) and the boundary conditions (4.2) - (4.4). 

If n+(Tinin) = ?i-(7'min), then in view of (3.32) Hb = Hb and the collection r turns into 

a Nevanlinna operator pair t G R{Hb) defined by (2.14) with Cj{X) G [Hb], A G C \ ]R, j G 
{0, 1}. In this case the boundary conditions (4.3)-(4.4) takes the form 

(4.5) Co(A)Fo62/ + Ci(A)rifcy = 0, A G C \ K. 

If in addition r G R°{Hb) is a selfadjoint operator pair (2.17) with Cj G [Hb], j S {0,1}, 
then (4.5) becomes a self-adjoint boundary condition (at the endpoint b): 

(4.6) CoTobV + CiTny = 0. 

Theorem 4.1. Let T be a symmetric relation in L\{I) defined by (3.39). Ifr = {t+,t_} G 
R{Hb,Hb) is a collection (2.4), then for every f G the boundary problem (4.1) - (4.4) 

has a unique solution y{t,X) = yf{t, A) and the equality 

(4.7) R{X)f = 7r{yf{;X)), J e L\{I), f & f, AgC\M, 
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defines a generalized resolvent i?(A) =: i?r(A) ofT. Conversely, for each generalized resol- 
vent -R(A) ofT there exists a unique r G R{'Hi,,'Hb) such that i?(A) = Rr{X). 

If in addition n+{Tjnin) = '^-(Tmin), then n+(T) = n_(T) and the above statements hold 
with Nevanlinna operator pairs t G R{T-Lh) of the form (2.14) and the boundary condition 
(4.5) in place of (4.3) and (4.4). Moreover, Rt{X) is a canonical resolvent ofT if and only 
if T € R^{Hb) is a self-adjoint operator pair (2.17), in which case Rt{X) = {T^ — \)~^ with 

(4.8) f ^ = {{y, /} e T^ax : Ti.y = 0, f^y = f^j/, CoT^^y + C^TuV = 0}. 

Proof Let 11 = {Hb © Hb.To,^!} be the boundary triplet (3.41) for T*. Applying to this 
triplet Theorem 2.17 we obtain the required statements. □ 

Remark 4.2. Let in Theorem 4.1 tq = {t+,t_} e R{l-Lb,Hb) be defined by (2.4) with 

(4.9) Co(A) = /^^, Ci(A) = 0, Do{\) = Pn,{&[Ub,Hb]), £'i(A) = 
and let i?o(A) := Rto{X) be the corresponding generalized resolvent of T. Then 

i?o(A) = (^0 - X)-\ A e C+ and i?o(A) = {A*^ - A)-\ A e C_, 

where Aq is given by (3.42). 

If in addition ?^+(Tmin) = '^^-(Tlnin); then tq turns into a self-adjoint operator pair tq = 
{{I-H,,0):nb} e R^iV-b) and i?o(A) = (Aq - A)"! is a canonical resolvent ofT. 

Proposition 4.3. Let the assumptions (Al) and (A2) be satisfied. Then: 

(1) For every A e C \ K there exists a unique operator solution vo{-,X) € £^[/fo,EI] of 
Eq. (3.3) such that 

(4.10) ri,t.o(A) = -Pff, i{fa-fb)vQ{\)=Pfj, AgC\K 

(4.11) TobVo{\) ={).: A e C+; Pn,TobVo{X) = 0, A e C_ 

(2) For every A € C+ (A G C_) there exists a unique operator solution ■u+(-,A) € 
C\[nb,M\ (resp. M-(-,A) ^ C\[nbM) of Eq. (3.3) such that 

(4.12) riau±(A) = o, i(ra-fb)u±{\)^{), Aec±, 

(4.13) ro6M+(A) =/^^, AGC+; P«,ro5H_(A) = A € C_. 

/n formulas (4.10)- (4.13) vq{X) and u±{X) are linear mappings from Lemma 3.3, (1) cor- 
responding to the solutions t;o(',A) and u±{-,X) respectively. 

Proof. Let U be the J-unitary extension (3.20) of U, let Foo be the operator (3.23) and let 

n = {'Hoffi'Hi, To, Fi} be the decomposing boundary triplet (3.37), (3.38) for T^ax- Assume 
also that 7±(-) are the 7-fields of 11. Since the quotient mapping tt isomorphically maps M\ 
onto ^^^(Tmin), it follows that for every A € C+ (A e C_) there exists an isomorphism 
Z+{X) -.Uo^Nx (resp. Z_(A) : Ui Nx) such that 

(4.14) 7+(A) =7rZ+(A), A G C+; 7_(A) = 7rZ_(A), A G C_. 
Let Fq and F'^ be the linear mappings given by 

r^, = (^~^^« " ^""l^ : domTma. ^Ho® Hb, 

F'l = (^^0" + fJT"^ + ^"^^ : domTn^a. ^Ho^Hb. 
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Then by (3.37) and (3.38) one has Tjlny, Xny} = Tj-y, y e j S {0,1}. Combining of 
this equahty with (4.14) and (2.24) gives 

(4.16) T'„Z+{\) = In„, AeC+; PhmT'qZ-W = lur, A e C_, 
which in view of (4.15) can be written as 

(4.17) (-r.. - f")) = ('»" j , A e c+ 

It follows from (4.17) and (4.18) that 

(4.19) ri„z±(A) = (-Pff, 0), i(r, - fb)z±{X) = (-fP^, o), a e c± 

(4.20) robZ+(A) = (0, J^J, AGC+; P«,rofcZ_(A) = (0, /„J, A G C_. 
Assume now that the block representations of Z±(X) are 

(4.21) Z+{X) = {vo{X), u+{X)) : Ha (SUb^Nx, A G C+ 

(4.22) Z_{X) = {vo{X),u-{X)):Ho(B'Hb^Mx, A G C_ 

and let ?;o(-, A) G £|[Fo,H], h+(-, A) G £|['H6,H] and u-(-, A) G £^['Hb,H] be the operator 
solutions of Eq. (3.3) corresponding to wo(A), w+(A) and M-(A) respectively (see Lemma 
3.3). Then the representations (4.21) and (4.22) together with (4.19) and (4.20) yield the 
relations (4.10)-(4.13) for wo(-,A) and u±(-,A). 

To prove uniqueness of va{-,X) and M±(-,A) assume that vq{-,X) G LWHq,M\,u+{-,X) G 
CWUbM and U-(-, A) G /:^[H6,e] are other solutions of Eq. (3.3) satisfying (4.10)-(4.13). 
Then for each /ig G ffo? /ife € 'Hfc and /i6 G "Hb the functions yi = {vo{t, X) — Vo{t, X))ho, 2/2 = 
(u+(i, A) — u^{t, X))hb and 7/3 = (?i_(i, A) — u_(t, A))/ifc are solutions of the homogeneous 
boundary problem (4.1) ~ (4.4) with / = and Cj(A), Dj{X), j G {0,1}, defined by 
(4.9). Since by Theorem 4.1 such a problem has a unique solution j/ = 0, it follows that 
2/1=2/2 = 2/3 = and, consequently, vo = vq and u± =u±. □ 

Proposition 4.4. Assume the hypothesis (Al) (A3). Lei 11 = {"Ho © "Hi, Fq, Fi} 6e t/ie 

decomposing boundary triplet for T-a^ayi defined in Proposition 3.6, let 7±(-) and M±{-) be 
the corresponding j-field and the Weyl function, respectively. Then 7±(-) is connected with 
the solutions vo{-,X) and u±{-,X) from Proposition 4-3 by 

(4.23) j±{X) \ Ho=wvo{X), A G C±; 

(4.24) j+{X) \nb = 7ru+{X), A G C+; j-{X) \ Hb = ttu-{X), A G C_. 
and the block representations 

(4.25) M+(A) = Zll[^^:Ho®Hb^Ho®nb, A G C+ 

(4.26) M_ (A) = ( ^^^^l^ j^j) :Ho®nb^Ho®nb, A G C_ 
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hold with 



(4.27) mo{X) = {Toa+fa)vo{X) + ^PH, AeC\M 

(4.28) M2±(A) = (Toa + fa)u±iX), A e C± 

(4.29) M3+(A) = -ri6z;o(A), Mi+{X) = -Tuu+iX), A e C+ 

(4.30) M3_(A) = (-Fib + iPn^rob)vo{X), 

(4.31) M4-(A) = (-rifc + zP„^^ro6)w_(A), AeC_. 



Proof. The equalities (4.23) and (4.24) are immediate from (4.14) and (4.21), (4.22). 

Next assume that Fg and F'^ are the linear mappings (4.15) and let M±{-) have the block 
representations (4.25), (4.26). Then by using (4.14) and (2.22), (2.23) one obtains 

riZ+(A)=M+(A), AeC+; {r[ + iP^^r'o)Z_{X) = M_{X), AeC_, 

which can be represented as 



Hence 

(4.34) roa^±(A) = (Pffmo(A), PhM2±(A)), A e C± 

(4.35) i(f „ + f b)Z±(A) = (P5mo(A), PhM2±{X)), X G C±. 

(4.36) TibZ+iX) = (-M3+(A), -M4+(A)), A e C+, 

(4.37) (-Fib + iPn^Tob)Z,{X) = (M3_(A), M4-(A)), A e C_. 

Summing up the second equality in (4.19) with (4.34) and (4.35) one obtains 

(4.38) (Foa + f a)^±(A) - (mo(A) - M2±{X)), X G C±. 



Combining now (4.36)-(4.38) with the block representations (4.21) and (4.22) of Z±{X) we 
arrive at the equalities (4.27)-(4.31). □ 

In the case of equal deficiency indices the statements of Propositions 4.3 and 4.4 can be 
rather simplified. Namely the following corollary is obvious. 

Corollary 4.5. Let the assumptions (Al) and (A2) be saiisfied, n^{T^-^i^) = n_{T„^in), and 
let Aq be the selfadjoint extension of T^i^ given by (3.42). Then for every A G p{Aq) there 
exists a unique pair of operator-valued solutions vo{-, A) € £\ [Ho,M] and u{-, A) e C'^['Hb, H] 
of Eq. (3.3) satisfying the following boundary conditions: 

T,,vo{X) = -Ph, iifa-fb)vo{X) = Pg, TobVo{X) = 0, Xep{Ao), 

ri„w(A) = 0, i{fa-fbMX) = 0, robu{X) = In„ XGp{Ao). 

Assume, in addition, that the assumption (A3) is fulfilled and H = {%, Fq, Fi} is an ordinary 
decomposing boundary triplet (3.37), (3.38) forT^^x. Then the corresponding Weyl function 
M(-) admits a block matrix representation 

(4.39) M(A) = {^^^^^ -.HoQHb^Ho^Hb, A e p(Ao) 
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with the entries given by 

(4.40) mo(A) = (Foa + ra)^^o(A) + ^Pg, M^iX) = (Fo^ + f „)u(A), 

(4.41) M^{\) = -TuVo{\), M4(A) = -ri(,u(A), A e p(^o). 

Theorem 4.6. Let the assum,ptions (Al) and (A2) he fulfilled and let t = {r+,T_} € 
R{'Hb,'Hb) be a collection of operator pairs (2.4). Then for each A G C \ M there exists a 
unique operator solution Vr{-,X) € £^[/foiEI] of Eq. (3.3) satisfying the boundary conditions 

(4.42) TiaVr{X) = -PH, AeC\M, 

(4.43) i{fa-fj,)vr{X)^P}j, AeC\K, 

(4.44) Co(A)ro6t;^(A) + C7i(A)ri(,t;^(A)=0, A e C+, 

(4.45) Do{X)TobVr{X) + Di{X)TuVriX) = 0, A e C_ 

(here Pr and are the orthoprojectors in Hq onto H and H respectively and Vr{X) is the 

linear map from Lemma 3.3 corresponding to the solution Vt{-,X)). Moreover, Vt{-,X) is 
connected with the solutions fo(', A) and u±{-, A) from Proposition 4-3 via the equalities 

(4.46) Vr{t,X)^vo{t,X)-u+{t,X){T+iX) + M^+iX))-'M3+iX), A e C+ 

(4.47) f,(<,A) =t.o(i,A)-w_(t,A)(r;(A) + M4-(A))-iM3-(A), A e C_, 

in which M3±(-) and M4±{-) are the operator functions given by (4.29)-(4.31). 

If in addition n+{T^in) = n-{T^in), then r e R{%b) is given by (2.14) and the boundary 
conditions (4.44) and (4.45) take the form 

Co{X)VobVr{X) + Ci{X)V^bVr{X) = 0, A e C \ M. 

Proof. To prove the theorem it is sufficient to show that the equalities (4.46) and (4.47) 
define a unique solution Vr{-,X) € C\[Ho,M\ of Eq. (3.3) satisfying (4.42) (4.45). 

Let ri = {Hb © 'Hb,to,ti} be a boundary triplet (3.41) for T*. Then by Proposition 
2.10, (3) the corresponding Weyl function is M+{X) = Mi^{X) and according to [39] one has 
€ p(t+(A) + M4+(A)), A e C+, and G /5«(A) + M"4-(A)), A G Therefore for each 
A G C \ M the equalities (4.46) and (4.47) correctly define the solution Vr{-,X) G £^[iJo,H] 
ofEq. (3.3). 

Combining (4.46) and (4.47) with (4.10) and (4.12) one gets the equalities (4.42) and 
(4.43). To prove (4.44) and (4.45) we let T+(A) = (t+(A) + M^+{X))-^ , X G C+, and 
T_(A) = (r;(A) + M4-(A))-\ a G C_. Then 

(4.48) T+(A) = {{T+{X)h, (/ - Af4+(A)r+(A))/i} ■.h&Hb} 

and t;(A) = {{r_(A)/i,/i - M4_(A)T_ (A))/i} : h G Hb}, which in view of (2.11) yields 

(4.49) T_(A) = {{{-T_{X)-iPn^+iP^^MUX)T.{X))h, 

b b 

{-Pu, + Pu,Mi^{X)T_{X))h) : h G iib}. 
It follows from (4.11), (4.13) and (4.30), (4.31) that 

(4.50) robVo{X)=-iPn^Ms-{X), TuvoiX) ^ -Pn.M^-iX), A G C_ 

(4.51) robU-{X) = lH,-tPntM4-W, Tuu-iX) = -Pn,M4-{><), A G C_ 
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and the relations (4.46) and (4.47) with taking (4.11), (4.13), (4.29) and (4.50), (4.51) into 

account give 

TobVriX) = -T+{X)M3+iX), AeC+, 
TibVriX) = -{I - M4+(A)T+(A))M3+(A), A e C+, 
robVriX) = {-iPui - T-{X) + iP^^M4_(A)T_(A))M3_(A), A e C_, 
TuVr{X) = {-Pn, + Pn,M4_{X)T_{X))Ms-{X), A e C_ 
Hence by (4.48) and (4.49) one has 

(4.52) {TobVr(X)ho,TuVr{X)ho} eT±(X), ho € Ho, A e C±, 

which in view of the equahties (2.4) yields (4.44) and (4.45). 

Finally, one proves uniqueness of «,-(•, A) in the same way as uniqueness of vo{-,X) in 
Proposition 4.3. □ 

5. m-FUNCTIONS 

Let the assumptions (Al) and (A2) at the beginning of Section 4 be fulfilled. 

Definition 5.1. A boundary parameter r (at the endpoint 6) is a collection r = {r+,r_} 
of holomorphic operator pairs (2.4) belonging to the class R{7ib,'Hb)- 

In the case of equal deficiency indices n+(Tiiiin) = «-(Tmin) one has Hb = Hb and, 
therefore, a boundary parameter is an operator pair r G Rijib) of the form (2.14). 

Let in addition to (Al) and (A2) the assumption (A3) be satisfied, let r be a boundary 
parameter and let «,-(•, A) e £^[iIo,EI] be the corresponding operator solution of Eq. (3.3) 

defined in Theorem 4.6. 

Definition 5.2. The operator function m^(-) : C \ K. [Ho] defined by 

(5.1) mr{\) = {Toa+Ta)Vr{X) + ^P^, A G C \ M, 

will be called the m-function corresponding to the boundary parameter r or, equivalently, 
to the boundary value problem (4.1)-(4.4). 

If ?T.+ (Tniin) = n_(Ti„i„), then m,r{-) corresponds to the boundary value problem (4.1), 

(4.2) and (4.5). In this case the m-function mr{ ) will be called canonical if r G BP{T-Lb)- 

It follows from (4.42) that in^{-) satisfies the equality 

(5.2) ^;.,„(A) (^"-^rt^") ^^(^0 ^ {"^^^^-Ph^") ^ ^0 ^ e /f. 

It turns out that for a given operator U and a boundary parameter t the m-function m^(-) 
is defined up to a self-adjoint constant. More precisely, the following proposition holds. 

Proposition 5.3. Suppose that under the assumptions (Al) and (A2) 

■.H®H®H^H®H®H, jG{l,2} 

are two J-unitary extensions ofU and Tq^ : AC{T;M) H, j G {1,2}, are the mappings 
(3.23). Moreover, let t be a boundary parameter and let 

m«(A) = i4'a + ^a)vr{X) + fP^, A S C \ M, j e {1, 2} 
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be the corresponding m-functions. Then there exists an operator B = B* G [H] such that 
the equality 

(5.3) m^(A) =mW(A) + B, AeC\M, 
holds with the operator B = B* G [Hq] given by B = BPh- 

Proof. By using the equality U*JUj = J, j G {1,2}, one can easily prove that there exists 
B = B* G [H] such that U2 = XUi with 

X = \ I \:H®H®H^H®H®H. 
\0 I J 

Therefore the mappings Ta ^y = Ujy{a),y G AC(I;H), j G {1,2}, are connected by T^^ = 
Xrl^\ which in view of (3.22) gives 

J- Oa - 1 Oa - la- 

Now by using (4.42) one obtains 

m?\X) = {T^^]+T,)vr{X) + lPs = 

(rW + fa)Vr{X) + |Pg - BT,aVr{X) = mi^\X) + BPh, 

which proves (5.3). □ 

In the following proposition we show that the m-function mr{-) can be defined in a 
somewhat different way. 

Proposition 5.4. Let under the assumptions (Al) (A3) t be a boundary parameter at 
the endpoint b, let (pu{-,X){G [Ho,M]) be the operator solution defined by (3.27) and let 
V'(-,A)(e [i?o,EI]), A e C, be the operator solutions of Eq. (3.3) with the initial data 

(5.4) V'a(A)(= Ui^ia, A)) = -.Ho^Ho^H. 

Then there exists a unique operator function m{-) : C\M — [Hq] such that for every X G C\M 
the operator solution v{-,X) of Eq. (3.3) given by 

(5.5) v{t, A) := ^u{t, A)m(A) + ipit, A) 

belongs to £'^[Ho,M.] and satisfies the boundary conditions (4. 43) -(4. 45). Moreover, the 
equalities v{t, A) = Vrit, A) and m(A) = mr(A) are valid. 

Proof. Let mr(-) be the m-function in the sense of Definition 5.2 and let v{-, A), A G C\]R, be 
the solution of Eq. (3.3) given by (5.5) with m(A) = mr{X). Then in view of (3. 28), (5. 4) and 
(5.2) one has Va{X) = Ur,a(A) and, consequently, v{t,X) = Vr{t,X). Therefore by Theorem 
4.6 v{-, A) belongs to £^[i7o,lHI] and satisfies the boundary conditions (4.43)-(4.45) . Hence 
there exists an operator function m(A)(= rUriX)) with the desired properties. 

Assume now that the solution v{-,X) of Eq. (3.3) given by (5.5) with some m(A) belongs 
to £^[i?o,IHI] and satisfies the boundary conditions (4.43)-(4.45). Then in view of (3.28) 
and (5.4) riai;(A) = —Ph and according to Theorem 4.6 v{t,X) = Vr{t,X). Therefore 
m(A) = m^(A), which proves uniqueness of m(A). □ 

Description of all m-functions immediately in terms of the boundary parameter r is 
contained in the following theorem. 
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Theorem 5.5. Let the assumptions (A1)-(A3) be satisfied and let M±{-) he the operator 
functions given by (4.25) - (4.31) (that is, M±{-) are the Weyl functions of the decomposing 
boundary triplet n = {'Hq®%\,Tq,Ti}). Moreover, let tq = {t+,t_} he a boundary param- 
eter defined by (2.4) and the equality (4.9). Then mo(A) = mT-j,(A) and for every boundary 
parameter r = {t+, r_} defined by (2.4) the corresponding m-function TOt-(-) is of the form 

(5.6) mr{X) = mo(A) + M2+(A)(Co(A) - Ci(A)M4+(A))-iCi(A)M3+(A), A G C+ 

Proof. One can easily verify that vo{t,X) = Vro{t,X), where vq{-,X) G £|,[i?o,]3] is the 
solution of Eq. (3.3) defined in Proposition 4.3. This and the equality (4.27) imply that 
mo(A) = mr„{X). Next, applying the operator Foa to the equalities (4.46) and (4.47) 
with taking (4.27) and (4.28) into account one obtains 

(5.7) m,(A)=mo(A)-M2+(A)(r+(A)+M4+(A))-iM3+(A), A e C+, 

(5.8) m^(A)=mo(A)-M2_(A)(T;(A)+M4-(A))-iM3-(A), A e C_. 
Moreover, according to [36, Lemma 2.1] e p(Co(A) - Ci(A)M4+(A)), A e C+, and 

-(r+(A) + M4+(A))-i = (Co(A) - Ci(A)M4+(A))-iCi(A), A e C+, 

which together with (5.7) yields (5.6). □ 

The following corollary is immediate from Theorem 5.5. 

Corollary 5.6. Let under the assumptions (A1)-(A3) ri+(T,iii,i) = n_(Tmin) and let M(-) 
be the operator function given by (4.39) -(4.41) (so that M() is the Weyl function of the 
ordinary decomposing boundary triplet H = {■H,ro,ri} for Tmaxj- Moreover, let tq = 
{(^•Hb) 0); "^b} G R°{%b). Then mo (A) = mi-o(A) and for every boundary parameter t defined 
by (2.14) the corresponding m-function mT-(-) is 

(5.9) TO^(A) =mo(A)+M2(A)(C7o(A)-C7i(A)M4(A))-iC7i(A)M3(A), AgC\R. 

Proposition 5.7. The m-function mri-) is a Nevanlinna operator function such that the 
relation 

(5.10) (ImA)-^ •Imm^(A) > j v;{t,X)A{t)vrit,X)dt 

holds for all X G C+. If in addition n+{Tjnin) = n-{Tynin), then (5.10) holds for all X G C\M. 

Proof. It follows from (5.7) and (5.8) that the operator function mr{-) is holomorphic in 

C \ R. Next, the equality M^CX) = Af_(A) for the Weyl functions (4.25) and (4.26) implies 
that m*o(X) = mo (A), Af2*+(A) = M3_(A), M3%(A) = M2-(A) and M4%(A) = M4_(A). This 
and (5.7), (5.8) yield the equality m* (A) — mr{X), A G C \ M. Now it remains to show that 
mr(-) satisfies (5.10). 

Let T = {T-f,T_} G R{'Hb,'Hb) be a boundary parameter defined by (2.4). Assume that 
A G C+, ho G Hq and let y := VT{X)ho, so that y = y{t) = Vrit, X)ho, i G X. It follows from 
(3.21) that 

(5.11) {Jy{a),y{a)) = (Jr„t/,r„y) = -2iIm{T,,y,Toay) + ilKyW^ 

Applying now the Lagrange's identity (3.5) to {y,Xy} G T^ax and taking the equalities 

(5.11) and (3.30) into account one obtains 

(5.12) imA • {y,y)A = UW^byW^ - \\fay\\^) + imir,ay,roay)- 

(Im(riby,rofay)-i||F^^.rofey|n. 
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It follows from (4.43) that Fhy = Tatj + iPgho and, therefore, 

(5.13) Whyf - \\Tay\f = WPshof + 21m{fay,Pgho). 
According to (5.2) 

(5.14) ToaV = PHmr{X)ho, TiaV = -Pnho, 

(5.15) f„y = Pgm,(A)/io - |Pg/io 
and substitution of (5.15) to the right hand part of (5.13) yields 

(5.16) llffeyll^ - \\f^yf = 2lm{Pgmr{X)ho, Pgho). 
Moreover, by (5.14) one has 

(5.17) Im {TiaV, Toat/) = Im {PHmr{X)ho, Ph/io). 
Substituting now (5.16) and (5.17) to (5.12) we obtain 

(5.18) ImA. (y,y)A =Ini(m^(A)/io,/io) - (Im (Fibt/, Tofey) - ^WPn^TobyW"). 

It follows from (4.44) that {Toby^^iby} G '''+(A). Therefore according to [38, Proposition 
4.3] 

(5.19) im{ruy,Toby) - i||P„xro6y||' > o. 

Moreover, in view of (3.1) one has 

(5.20) {y,y)A = {{ j^v*{t,X)/\{t)vr{t,\) dt)hM- 

Combining now (5.19) and (5.20) with (5.18) we arrive at the relation (5.10). □ 

Corollary 5.8. For each boundary parameter r the following equality holds: 

(5.21) ipu{x,\)v*{x,X)-Vr{x,X)ipu{x,X)= J, a; el, AeC\R. 

Proof. Let f/ be a J-unitary extension (3.20) of U and let Yq{x,X){£. [H]) be the operator 
solution of Eq. (3.3) with Fo,a(A)(= UYo{a,X)) = lu- Then by the Lagrange's identity (3.5) 

one has 

Y^{x,X)JYo{x,X) = Y^{a,X)JYo{a,X) = U-^*JU-^ = J 
and, consequently, 

(5.22) Yo{x,X)JYo*{x,X) = J, xel, AeC\]R. 
Since by Proposition 5.7 m*(A) = mr{X), it follows from (5.2) that 

<„(A) = (m,(A) + fP^, -Ih) -.Ho^H^Ho. 
Combining of this equality with (3.28) yields 
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Now by using (5.22) one obtains 

ipu {x, \)v* (x, A) - Vr (x, X)(pIj (x, A) = 

Yo{x,X){^U,aWVr,a{W " Vr,aW^*U,aW)yO i^,^) = Yo{x, X)JYo* {x,X) = J. 

□ 

In the following proposition we show that a canonical m-function mr(-) is the Weyl 
function of some symmetric extension of Tmin (in the sense of Definition 2.9). 

Proposition 5.9. Let the assumptions (A1)-(A3) be satisfied and let n+{Tmin) = n-{Tmin)- 

Moreover, let t € R^{7ib) be a boundary parameter (2.17), let tv(-,A) G C'j^[Ho,'Si] be the 
operator solution of Eq. (3.3) defined in Theorem 4-6 and let mri-) be the corresponding 
m-function. Then: 

(1) The equalities 

T = {{y, /} G Ilnax : y{a) = 0, hy = 0, CoTobV + CiTny = 0}, 
T* = {{y, /} e T^ax : CoToby + C,T,by = 0} 

define a symmetric extension T of Tmin and its adjoint T* ; 

(2) The collection 11 = {iJo,ro,ri} with the operators 

(5.23) fo{y,/} = -ri„y + z(r„-f6)y, fi{y, /} = Toa?/ + i(f„ + fft)?/, {yj}^f\ 
is a boundary triplet for T* . Moreover, the field and Weyl function M {■) ofH are 

(5.24) 7(A) = 7rw^(A), M{X)=mr{X), A e C \ R. 

(3) The following identity holds 

(5.25) mr{n)-m;{X) = {ii-X) j v;{t,X)A{t)vr{t,n)dt, ii,XeC\R. 

This implies that for the canonical m-function m-ri-) the inequality (5.10) turns into the 

eqluality, which holds for aZ/ A G C \ M. 

Proof. Clearly, we may assume that r is given in the normalized form (2.18), in which case 
the operators 

(5.26) ro{^, /} = {-Tiay + iiTa - fb)y, CoToby + CiTiby}{€ Ho 8 Hb), 

(5.27) ri{^, /} = {Toay + |(r„ + fb)y, C.Toby - CoT,by}{& Ho e Hb) 

{{y,f} e Tniax) form a decomposing boundary triplet 11 = {■H,ro,ri} for Tmax with H = 
Ho®'Hb- Let 7(A) be the 7-fiold and 

(5.28) ^(A)=(§;f,5 ^f^^:Ho®Ub^Ho®Hb, AeC\K, 

be the Weyl function of the triplet 11. Assume also that vo{-,X) G C\[Ho, H] is the operator 
solution of Eq. (3.3) defined in Proposition 4.3 (for the triplet 11). Then uo(i, A) = Vr{t, A) 
and (4.23) yields 7(A) \ Hq = irVriX). Moreover, in view of (4.40) one has mo (A) = 
mx(A), A e C\M. Applying now Proposition 2.10 to the triplet n (with Hq = Hi = Hq) we 
obtain statements (1) and (2). Finally, (5.25) follows from the identity (2.26) for the triplet 
n and Lemma 3.3, (2) applied to the solution Vr{-,X). □ 
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Remark 5.10. Let under the assumptions (Al) (A3) 11 = {?^,ro,ri} be an ordinary de- 
composing boundary triplet for Tmax, let t G i?°('Hb) be a boundary parameter given in the 
normalized form (2.17), (2.18) and let 11 = {■H,ro,ri} be another decomposing boundary 
triplet for Tmax defined by (5.26) and (5.27). The triplets 11 and 11 are connected by 

To \ I Xi X2 \ / To 



where Xj E [Hq ® Hb] are defined as follows 



CoJ ' ^ \p -CiJ ' \0 CiJ ' ^ ^0 

Therefore according to [10] the Weyl functions M(-) and M(-) of the triplets 11 and 11 
respectively are connected by means of linear fractional transformation, 

(5.29) M(A) = {X3 + XiM{\)){Xi + X2M{X))-\ 

By using the block representation (4.39) of M(A) one obtains 



(X,+X.M(A))-=(_^^^^ Co-W 



/ 

and (5.29), (5.28) imply that mo(A) coincides with the right-hand side of (5.9). This and 
the equality mx(A) = mo(A) obtained in the proof of Proposition 5.9 yield (5.9). Thus, for 
canonical m-functions mx(-) formula (5.9) is a simple consequence of the relation (5.29) for 
Weyl functions. 

Note that in this proof we follow the reasonings of [10, Remark 86], where the Krein 
formula for canonical resolvents was proved in a similar way. 

6. Spectral functions 

6.1. Green's function. In the sequel we put := L\{I) and denote by ijf, the set of all 

f € S) with the following property: there exists j3 £l (depending on /) such that for some 
(and hence for all) function / S / the equality A{t)f{t) = holds a.e. on {(3,b). 

Assume hypothesis (Al) and (A2). Let (pu{-,X) be the operator-valued solution (3.27), 
let r be a boundary parameter and let Vr{-,X) € £^[ifo5lHI] be the operator- valued solution 
of Eq. (3.3) defined in Theorem 4.6. 

Definition 6.1. The operator function Gt(-, •, A) : X x X — >■ [H] given by 

(6.1) G.(a;,i,A) = l'^^f'^l''^!''-!' , AeC\M 

will be called the Green's function corresponding to the boundary parameter r. 

Next we compute the generalized resolvent of T in terms of the Green's function. 

Theorem 6.2. Let r be a boundary parameter and let Rr{-) be the corresponding generalized 
resolvent of the relation T (see Theorem 4-1)- Then 

(6.2) Rr{X)f = n(^j^Gr{;t,X)A{t)f{t)dty f G Sj, f G f. 
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Proof. Since Vr{-,X) € jC%[Ho,'a], it follows from (6.1) that 

\Gr{x,t,X)Ai{t)\f dt <oo, x€l. 
Hence for each / e £\ (I) and x e X one has 

\Gr{x,t,\)A{t)f{t)\\dt< [ \\Gr{x,t,\)A^{t)\\-\\A^{t)f{t)\\dt<^ 



L 



IX 

•^2 



and, therefore, the equality 

(6.3) yf=yf{x,X):=J^Gr{x,t,X)A{t)f{t)dt, f e C\{I), AeC\M 

correctly defines the function yf{-, ■) : X x C \ M — )• H. This implies that (6.2) is equivalent 
to the following statement: for each f E ^ 

(6.4) yf{-,X)&£l{I) and Rr{X)f = n{yf{-,X)), f e f, AeC\R. 

To prove (6.4) first assume that / G S)b- We show that in this case the function j//(-,A) 
given by (6.3) is a solution of the boundary problem (4.1)-(4.4). It follows from (6.1) that 

(6.5) yf = yf{x, X) = >Pu{x, X)Ci{x, X) + Vr{x, X)C2{x, X) = Y{x, X)G{x, A), 
where 

Ci{x,X)= fv;{t,X)A{t)f{t)dt, G2{x,X)= f\l,it,X)A{t)f{t)dt 

Y{x,X) = ((^„(x,A), «,(.T,A)), C{x,X) = {Ci(,x,A),C2(.T,A)}(e Ho® Ho). 
Moreover, by (6.5) and the equality A{t)f{t) — (a.e. on (/3, b)) one has 

(6.6) yf{x,X)=Vr{x,X) J^ip*u{t,X)A{t)f{t)dt, x € i/3,b). 

This implies that y/ e AC(I;H) n £i(X). Next, in view of (5.21) one has 

Y{x,X)C'{x,X) = {-Mx,XKix,X)+Vr{x,X)>fi*uix,X))A{x)f{x) = 

-JA{x)f{x). 

By using this equality we obtain 

Jy'f{x, X) - B{x)yf{x, A) = {JY\x, A) - B{x)Y{x, X))C{x, A)+ 

JY{x, X)C'{x, A) XA(x)Y{x, X)C{x, A) - J^A{x)f{x) = 

XAix)yf{x,X)+A{x)f{x). 

Thus, for each A € C \ M the function yf{-, A) satisfies (4.1) a.e. on X. 

Next we show that yf{-,X) satisfies the boundary conditions (4.2) (4.4). Let f7 be a 
J-unitary extension (3.20) of U and let be the mapping (3.21). Since by (6.5) 

r„y/ = ^u,aW J^Kit,x)Mt)f{t)dt, 

it follows from (3.28) that 

(6.7) f^vf =PqJ v;it,X)A{t)f{t)dt = J {vr{t,X) \ HrAit)f{t)dt, 

(6.8) T^aVf = 0. 
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Moreover, according to Remark 3.5, (1) the equality (6.6) yields 



(6.9) 



Tbyf = fbVr{X) [ <p*u{t,X)A{t)f{t)dt, 



(6.10) 



Ji 



(6.11) 



Ji 



In view of (6.8) the first condition in (4.2) is fulfilled. Next, by (4.43) and (5.2) one has 



(6.12) %Vr{X) = faVr{X) + iP^ = (Pgm^(A) - iPg) + iPq = PfjimriX) + ^Ih,). 



(here we make use of the relation ?7i*(A) = m^(A)). Hence the second condition in (4.2) is 
fulfilled. Finally combining (6.10) and (6.11) with (4.44) and (4.45) one obtains the relations 
(4.3) and (4.4) for yf. Thus yf{-,X) is a solution of the boundary problem (4.1) (4.4) and 
by Theorem 4.1 the relations (6.4) hold. 

Now assume that / G is arbitrary, / G /, and yf = yf(x,X) is given by (6.3). Assume 
also that /„ = /X[a f,-i]> fn = 7r/n(e -Qb) and let = yf^{x,X) be given by (6.3) with 
fn{t) in place of f{t). Moreover, let a function yn G be such that nyn = i?i-(A)/. 

Since fn ^ f and ttj//^ = i?^(A)/„, it follows that \\yR — y/„||A — >■ 0. On the other hand, 
yf^{x,X) — > yf{x,X), x G T, and, consequently, A{x){yf{x,X) — yij{x,X)) = a.e. on X. 
Hence y/ G jO.\{I) and ny/ = nyn = i?x(A)/, which gives the relations (6.4) for /. □ 

Remark 6.3. Theorem 6.2 generalizes several results in this direction. More precisely, in the 
case of Hamiltonian system (3.2) {H = {0}) and separated boundary conditions formulas 
(6.1) and (6.2) for canonical resolvents of T^in were proved in [20, 30]. Moreover, assuming 
that the minimal operator Tmin is generated by Hamiltonian system with the minimal defi- 
ciency indices n±(Ti„in) = dim if, formulas (6.1) and (6.2) for generalized resolvents of Tmin 
have been obtained in [11, 12]. Note also that formulas for canonical and generalized resol- 
vents of even order ordinary differential equations subject to separated boundary conditions 
are known as late as the middle of nineteenth (see e.g. [13, 43, 46]). 



Moreover, in view of (5.2) 




and (3.28) gives Vr{t,X) \ H = ipu{t,X){mr{X) 
(6.9), (6.12) and (6.7) one obtains 



^Iho) \ H. Combining this equality with 
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6.2. The space L^(S;'H). Let be a finite dimensional Hilbert space. 

Definition 6.4. A non-dccrcasing operator function S(-) : R — >■ [H] is called a distribution 

function if it is left continuous and satisfies the equality S(0) = 0. 

Next recall the definition of the space LF'{T,;'H) (see e.g. [43, Section 20.5], [5, Section 
7.2.3]). Denote by Co{H) the set of continuous vector functions f iR^'H having compact 
supports. Introduce the semi-scalar product on Co{'H) by setting 

(6.13) {f,g)m^;n) = / {dmf{t),9{t)n) = Jim V(E(Afe)/(a),5(a))- 

Js. k^l 

Here Wn = {a = to < ti < ■ ■ ■ < tn = b} denotes a partition of a segment [a, b] containing 
the supports of functions / and g, (i(7r„) is the diameter of the partition 7r„, S(Afe) := 
T,[tk) — Ti{tk-i), and ^fe G [tk-i,tk]- The limit in (6.13) is understood in the same sense 
as in the definition of the Riemann-Syieltjes integral, i.e., a particular choice of 7r„ with a 
given diameter and of € [tk-i,tk] is irrelevant. 

The completion of Cq{H) with respect to the semi-norm p{f) (/, /))£2(2-h) gives rise 
to a semi-Hilbert space L^(Y,,H) (i.e., to a complete space with a semi-norm in place of 
norm). Denoting by kerp := {/ e L^(E,iJ) : p{f ) = 0} the kernel of the semi-norm, we 
introduce the quotient space L^(I];'H) := L^{Ti, H)/k.erp which is already Hilbert space. 

Let E = {(Jij)'^j^i be a matrix valued measure generated by a distribution function S(-) 
and let a = (Jjj. Clearly, the measure E(-) is absolutely continuous with respect to a 
(in fact both measures are equivalent). Therefore, by the Radon-Nykodim theorem, there 
exists a a-measurable matrix density ^'(•) = {'4'ij{'))7,j=i ^^ch that 

S(5) = j^^{t)da{t), ^{t) := (^,,(t))i:,=i = {dcjii/da)l^^^, S e ^^(R). 
Let Lq{T,,C"') be the set of cr-measurable vector- valued functions / : R C" satisfying 

(6.14) I I/I l|2(s,c") ■= X^*^*)-^^*)' '^^^^^ < 

Theorem 6.5. [24] The spaces i^(S,C") awe? L^(E,C") are identified isom,etrically with 
the spaces Z§(I],C") anrf L§(S,C") := Zg(S, C")/A^o, respectively, where Nq^= {/ e 

semi-norm. Therefore, f € L^(S,C") 
and only if f is a-measurable and the norm (6.14) is finite. 

It was shown in [35] that the spaces L^(S,C") and L^(E,C") admit the representation 
in the form of direct integrals 

(6.15) Z2(S,C")= / ®G{t)da{t), L'^{E,C'')= [ ®G{t)da{t), 

where G(t)is the n-dimensional Euclidian space with the semi-scalar product (/, g) := 
{^{t)f,g) and G{t) = G{t)/{f G G{t) : (*(t)/,/) = 0}. In particular, representation 
(6.15) gives a simple proof of Theorem 6.5 (as distinguished from the known proofs in [24] 
and [13]). 
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6.3. Spectral functions and the Fourier transform. Let the assumptions (Al) and 
(A2) at the beginning of Section 4 be satisfied and let ipu{',X){G [Ho,M]) be the operator 
solution of Eq. (3.3) with the initial data (3.27). For each f G Sjb introduce the Fourier 
transform /(•) : M ^ iJo by setting 

(6.16) f{s) = l^^tjit,s)A{t)f{t)dt, fGf. 

Note that /(•) is uniquely defined by /. i.e., it does not depend on the choice of / € /. 

Next assume that r = {T+,r_} e R{'Hb,'Hb) is a boundary parameter given by (2.4) 
(with Ho — 'Hb and "Hi ~ Hb)- Then according to Theorem 4.1 the corresponding boundary 
problem (4.1)-(4.4) generates the generalized resolvent 

(6.17) Rr{X) = Pf,if^ - X)-^ \ Sj, AeC\]R, 

of the symmetric relation T G C{^)). The equality (6.17) uniquely (up to the unitary 
equivalence) defines a self- adjoint ^-minimal relation T"^ in ^ D ^ such that T C T"^. 
Denote also by fV(") the corresponding spectral function of T, so that in view of (2.29) 



3.18) RriX)= [ AG 



\1 



In the following wc fix some J-unitary extension U oi U (see (3.20)) and denote by TOt-(-) 
the TO-function of the boundary problem (4.1)-(4.4). Note that in view of Proposition 5.3 a 
choice of U does not matter in our further considerations. 

Definition 6.6. A distribution function E(-) = Et-(-) : K — ^ [Hq] is called a spectral function 
of the boundary problem (4.1)-(4.4) if, for each f G Sjb and for each finite interval [a, (5) C M, 
the Fourier transform (6.16) satisfies the equality 

(6.19) {{Fr{p)-Fr{a))fJ)s,= [ (dS.(s)/(s), /(s)). 

Note that the integral on the right-hand side of (6.19) exists, since the function /(•) is 
continuous (and even holomorphic) on R; moreover, by (6.19) / G I/^(Ex; Hq). 
Let ^0 := ^ 6 mulT"^, so that 

(6.20) ^=^oeniulf^ 
Then by (2.30) and (6.19) one has 

(6.21) \\P^J\\^ = \\f\\LHi:M f^^b 

and, consequently, ||/|| < ||/||. Therefore for each f G Sj there exists a function / G 
L^{'Er\Ho) (the Fourier transform of /) such that 

limll/-/ <^^(i,.)A(t)/(t)di|U.(s,.^„)=0, /G/, 

and the equality Vf = /, / € -fl, defines the linear operator V : L^(Et-; Hq) such that 

(6.22) \\Vf\\LHi:..,„,) = \\P^J\\^, f € Sj. 
This implies that V is a contraction from Sj to L^^Sr', Ho)- 
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Theorem 6.7. For each boundary parameter t there exists a unique spectral function S^(-) 
of the boundary problem (4.1) -(4.4). This function is defined by the Stieltjes inversion 
formula 

1 f'~^ 

(6.23) T,t(s) = lim lim — / Im rrir ( <J + is) da. 

i5->-+0e^-+0 TT J_g 

Proof. It follows from Proposition 5.4 that the Green function (6.1) admits the representa- 
tion 



Gr{x,t,X) = (pu{x,X)mr{X)(Pu{t,X) + Go{x,t,X), 

ijj{x, A) (filjit, A), X > t 
ipu{x,X)'ip*{t,X), x<t 



where Go{x,t,X) = 



Now by using (6.2) and the Stieltjes-Livsic inversion formula one proves the theorem in the 

same way as Theorem 4 in [46]. □ 

Next, similarly to [13, 43, 46] one can prove the following theorem. 

Theorem 6.8. Let V : ^ L'^(T,t-; Hq) be the Fourier transform corresponding to the 
spectral function St(') (ind let V* be the operator adjoint to V. Then for each function 
g = g{s) e Z/^(S^;ifo) with the compact support the function 

fg{t) := / ^u{t,s)d11r{s)g{s) 
Jm. 

belongs to C\il) and V*g = wfg. Therefore 

(6.24) V*g = TrQ^^ui-,s)dJ:r{s)g{s)y 5 = ^(s) S L'(I],; ffo), 

where the integral converges in the semi-norm (3.1). 

For a spectral function St(") denote by A the multiplication operator in L^(I]^; Hq) given 
by the relations 

domA = {/ e l^j:^; Ho) tf{t) e l\y.,-Ho)}, 

^ ' {Af){t) =tf{t), /edomA. 

As is known A is a self-adjoint operator and the spectral measure E\{-) of A is 

(6.26) iEAi6)f)it)^X5{t)f{t), feL\^^;Ho), S e 

where xs{') is the indicator of the Borel set S. Moreover, in view of (6.19) one has 

(6.27) Fr{^) - Fr{a) = V* E^iia, p))V, [a,/3) C M. 

Proposition 6.9. Let St(-) be a spectral function of the boundary problem (4.1)- (4.4), let 
V : S) ^ L^{Yjt;Ho) be the corresponding Fourier transform and let 

(6.28) Lo := clos (V^) 

Then the operator A is Lo-minimal (in the sense of Definition 2.12). 
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Proof. Let Li := kerF*(= L'^{Y^r] Ha)QLa) and let ,g <E Li be a vector such that _Ea([Q!, /3)).9 G 
Li for each bounded interval [a, [3) C M. Then AE^aCIq;, € Li and, consequently, 
y*^A([a!,^))S' = and V*AE^{[a/p))g = 0. This and (6.24) imply that the functions 

(6.29) y{t)= I ^u{t,s)dJ:ris)g{s), f{t)= [ ^u{t,s)dJ:r{s)sgis) 
satisfy the equalities 

(6.30) ^{t)yit) = and A{t)f{t) = a.e. on I. 
On the other hand, in view of (6.29) one has 

Jy'{t) - B{t)y{t) = / {J^'uit, s) - B{t)ipu{t, s)) dS,(s)5(s) = 
I {sA{t)ipu{t, s)) dS,(s)ff(s) = A{t)f{t). 

J[aJ3) 

Combining this equality with (6.30) and taking definiteness of the system (3.2) into account 
one gets 

(6.31) y{t)= I ipu{t,s)dY.r{s)g{s)={), tGl, [a,(3)cR. 

It follows from (3.27) and (3.28) that the operator (pu(a,s) dos not depend on s and 
kenpu{a,s) = {0}. This and (6.31) yield 

/ dS^(s)5(s) = 0, [a,^)cM, 

which gives the equality 5 = 0. Thus the condition (2) of Definition 2.12 is satisfied. □ 

Let be decomposed as in (6.20) and let 

(6.32) fly:=^onfl, flfc:=muir^nfl, flc = G (i3y © flfc). 
Then 

(6.33) ^=S)v®^k®^c 

and by (6.21) the operator V (the Fourier transform) is isometric on S)v^ strictly contractive 
on and has S)k as a kernel. Observe also that mulT C ^3^, so that V \ mulT = 0. 
Next assume that ^q:= S^Q mulT, so that can be represented as 

(6.34) i5 =i3o©mulT. 

It follows from (6.33) that ijo is the maximally possible subspace of on which the Fourier 
transform V may be isometric. 

Definition 6.10. A spectral function 5]^(-) of the boundary problem (4.1)-(4.4) will be 

referred to the class SFq if the operator 

(6.35) Vo:=V\ 
is an isometry from Sjq to Z/^(E^; Hq). 
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By using ij-minimality of T'^ one can easily show that 

(6.36) S^(-) e SFo ^ mulf^ = mulT. 

Therefore aU spectral functions belong to SFq if and only if mul T — mul T* . 
If St(') G SFq, then by (6.24) for each / G Sjq the inverse Fourier transform is 

(6.37) f = 7r(^j^^u{;s)di:r{s)m^ 

Theorem 6.11. Let t be a boundary parameter, let Sr(") be the spectral function of the 
boundary problem, (4.1) (4.4) and let V be the corresponding Fourier transform,. Assume 
also that e C(io) is the (exit space) self-adjoint extension ofT defined by (6.17), ioo C io 
and ijo C are the subspaces from decompositions (6.20) and (6.34) respectively and T'^ is 
the operator part of T'^ (so that T'^ is a self-adjoint operator in Sjo). //St-(-) G SFq, then 
Sjo C Sjo and there exists a unitary operator V G [Sjo, L'^{^r', Hq)] such that V \ S)o = K)(= 
V \ Sjo) and the operators T'^ and A are unitarily equivalent by means of V . 

Moreover, if m\AT = mulT* (th,a,t is, the condition (CI) in Remark 3.9 is fulfilled), then 
the statements of the theorem hold for each spectral function St(')- 

Proof. Since in view of (6.36) mulT^ = mulT, it follows that C i5o and the decomposi- 
tion (6.20) takes the form 

(6.38) ^=^o©mulT. 

It follows from (6.27) and (2.28) that for each finite interval [a, ^) C R the spectral function 
Et{-) of satisfies the equality 

P^Ma,p)) \ Sjo = Vo*E4[a,p))Vo = Vo*{PLMa, p)) \ Lo)Vo, 

where Lq = VoS)o{= VS^o). This implies that 

(6.39) Psi„{T--X)-^\Sjo = Vo*{PLo{A-X)-^\Lo)Vo, AeC\R. 

Since T'^ is i^-minimal, it follows from (6.34) and (6.38) that the operator T"^ is i^o-minimal. 
Moreover, according to Proposition 6.9 the operator A is Lo-minimal. Now, applying Propo- 
sition 2.14 to operators and A we arrive at the desired statements for Sr(') € SFq. 

The last statement of the theorem follows from the fact that in the case mul T = mul T* 
the inclusion E^(-) G SFq holds for each spectral function St(')- ^ 

Combining of Theorems 6.11 and 4.1 yields the following corollary. 

Corollary 6.12. Let t be a boundary parameter and let St(') be the spectral function of 
the boundary problem (4.1) -(4.4). Then the following statements are equivalent: 

(1) n+(Ti„m) = n-^T^in), t G BP{(Hb) and the canonical self-adjoint extension T"^ of T 
given by (4.8) satisfies the equality mulT"^ — mulT 

(2) The Fourier transform V isometrically maps Sjo onto L^{Y.r'.Ho) (that is, V \ Sjo is 
a unitary operator). 

If the statement (1) (and hence (2)) is valid, then the operator (the self-adjoint part 
of T'^) and the multiplication operator A are unitarily equivalent by means of V. 

Theorem 6.13. Assume that T is a densely defined operator, that is, the condition (C2) in 
Remark 3. 9 is fulfilled. Then for each boundary parameter t and the corresponding spectral 
function St-(-) the following hold: (i) T'^ is an operator, that is, T'^ = T'^ ; (ii) the Fourier 
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transform V is an isom,etry; (iii) there exists a unitary operator V G [S), L'^(Y,t-; Hq)] such 
that V \ = V and the operators T'^ and A are unitarily equivalent by means of V. 
Moreover, the following statements are equivalent: 

(1) n+(T,„in) = n-(T,„in) and r G R^{'Hb), so that T'^ is the canonical self-adjoint exten- 
sion ofT given by the boundary conditions (4.8); 

(2) VSj = L'^ {T,t', Hq) , that is the fourier transform V is a unitary operator. 

If the statement (1) (and hence {2)) is valid, then the operators T'^ and A are unitarily 
equivalent by means of V. 

Proof Since mulT = mulT* = {0}, the required statements are implied by Theorem 6.11 
and Corollary 6.12. □ 



It follows from Theorem 6.11 that the operators T"^ and A have the same spectral prop- 
erties. This implies, in particular, the following corollary. 

Corollary 6.14. (1) If t is a boundary parameter such that St-(-) e SFq, then the spectral 
multiplicity of the operator T"^ does not exceed V-{— dimiJo)- 

(2) If the condition (CI) in Remark 3.9 is fulfilled, then the above statement on the spectral 
multiplicity ofT^ holds for each boundary parameter r. 

In the next theorem we give a parametrization of all spectral functions St(") in terms of 

a boundary parameter r. 

Theorem 6.15. Let n_|_(rmin) = n_(rmin) and let M(-) be the operator function defined by 

(4.39) (4.41). Then, for each boundary parameter t G R{'Hh) given by (2.14) the equality 

(6.40) mr{\) = mo(A) + M2(A)(Co(A) - Ci{X)Mi{\))-^Ci{X)M^{\), A € C \ M, 

together with (6.23) defines a (unique) spectral function of the boundary problem (4.1)- 

(4.4). Moreover, the following hold: 

(1) St-(-) e SFq if and only if the following two conditions are satisfied: 

(6.41) lim -(Co(iy) - C^{iy)MSy))-^Ci{iy) = 0, 

(6.42) lim -MSv){CQ{iy) - Ci{iy)Mi{iy))-^Co{iy) = 0. 

y-s-oo y 

(2) Each spectral function St(-) belongs to the class SFq if and only if 

lim -M4{iy) = and lim ylm{M4{iy)h,h) = -\-oo, h G Hb, h ^ 0. 

y-s-oo y y^oo 

Proof. The main statement of the theorem directly follows from Corollary 5.6 and Theorem 
6.7. 

Next, consider the boundary triplet 11 — {'H6,ro,ri} for T* defined in Proposition 3.8. 
Since M(-) is the Weyl function of the decomposing boundary triplet (3.37), (3.38)for Tmax, 
it follows from Proposition 2.10, (3) that the Weyl function of the triplet n coincides with 
M4(A). Now applying to the boundary triplet n the results of [10, 7] we obtain statements 
(1) and (2). □ 
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6.4. The case of minimal equal deficiency indices. In this subsection we reformulate 
the above results for the simplest case of minimally possible equal deficiency indices of Tmin, 
which in view of (3.14) are 

(6.43) n+ (T^in) = n- {T^in) = v- ■ 

In this case z^f,_ — 0, i'b+ = v and according to Lemma 3.4 there exists a surjective linear 
mapping F;, : domT^ax — >■ H such that 

(6.44) [y,z]b^i{TbV,Tbz), j/, 2: € domT^ax- 

Below we suppose that the assumption (Al) at the beginning of Section 4 is fulfilled and 
that Fb is a surjective operator satisfying (6.44). 

It follows from Proposition 3.8 and Theorem 4.1 that the equality 

T = {{y, /} e Tlnax : Tiav = 0, T^y = hy} 

defines a self-adjoint relation T in Sj{= i^(X)) and the (canonical) resolvent of T is given 
by the boundary value problem 

(6.45) Jy' -B(t)y = XA(t)y + A{t)f{t), tGl, 

(6.46) Tiay-O, f ay = f 52/, AeC\R. 

Next, in view of Theorem 4.6 for each A S C \ M there exists a unique operator solution 
v{-,X) e £|,[ifo,H] of Eq. (3.3) such that 

(6.47) Tiav{X) = -PH, i{fa-fb)v{X)=Pjj, AeC\K. 

Moreover, if J7 is a J-unitary extension (3.20) of U and Foo is the mapping (3.23), then the 
(canonical) m-function m(-) of the problem (6.45), (6.46) is given by the equality 

(6.48) m{X) = iToa+faHX) + ^PQ, AeC\K, 
or, equivalently, by the relations 

v{t,X) ■.^^uit,^)mi^}+^{t,X) e£l[Ho,m], i{f,-fb)viX) = P^, AeC\M. 

The boundary problem (6.45), (6.46) has a unique spectral function S(-), which is defined 
by the Stieltjes formula (6.23) with TOr(-) = m{-). Moreover , Corollary 6.12 implies that the 
corresponding Fourier transform V isometrically maps Sjo{= ^ Q mulT) onto i^(S; Hq). 

6.5. Example. In this subsection we provide an example illustrating the results of the 
paper. 

Let X = [0, oo) and let i5(-) be a Borel function on X such that S{t) > (a.e. in I) and 

/•oo 

C := 6{t)dt < 00. 
Jo 



Assume also that in formulas (3.10) and (3.12) H = H = C, so that H = and Hq = C^. 
Consider the symmetric system 

(6.49) Jy' ^A{t)f{t), teX, f e £l{X), 

where J and A(t) arc given by the matrices 

/O -1\ / ^{6{t) + 1) liSit) - 1) 

J = i , A{t) = 1 

Vl 0/ y_|(^(t)_i) i((5(0 + l) 
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Clearly, A(t) is a nonnegative invertible matrix (a.e. on T); therefore the system (6.49) is 
definite and T^in is a densely defined operator in L^(X). The immediate checking shows 
that the homogeneous system 

(6.50) Jy' = XA{t)y, teX, A e C, 
has a fundamental solution 

(6.51) Y{t,X) = 

where ^ 

$(i) := / 5{s)ds. 
Jo 

Denote by y^^\-,X), y^'^\-,X) and 2/(^^(-,A) vector solutions of Eq. (6.50) formed by 
the first, second and third columns of the matrix (6.51) respectively. It is easily seen that 
2/«(.,A), y(^H;X) e £i(J), y(^H;X) i L\{X) for all A e C+ and 2/«(-,A), y^^^X-.X) e 
C\{I), y'^^\-,X) ^ i^\{X) for all A £ C_. Therefore the operator Tmin has minimally 
possible equal deficiency indices n+(Tmin) = n_(rmin) = 2. 

Let 9{-) e be the solution of Eq. (6.50) given by 

0{t) ^ ;^e-V^^(M) - ^e-^{e*W,0,-ie*«}. 

Since [9,9]oo = i, it follows from Remark 3.5, (2) that the equality T^y = [y,0]oa, y G 
domT^ax, defines the surjective linear mapping Th : domT^ax — C satisfying (6.44). 

We assume that U = I (see (3.20)). Then for each function y e domT^ax decomposed as 

y{t) = {yo{t),y{t), yi{t)}{€ C 8 C 8 C), tel, 

one has Foa?/ = ?/o(0), ^aV = y(0)) ^laU = 2/1 (0) and the boundary problem (6.45), (6.46) 
can be written as 

(6.52) Jy' = XA{t)y + A{t)f{t), t € X, 

(6.53) yi(0) = 0, y{0) = [y,6]b, AeC\M. 
According to Subsection 6.4 there exists a unique operator solution 

fro{t,X) qo{t,X}\ 

(6.54) v{t, A) = ?{t, A) q{t,X) : C © C 

\nit,X) qi{t,X)J 

of Eq. (6.50) belonging to £^[i?o,H] and satisfying the boundary conditions (6.47), which 
in our case take the form 

ri(0,A) = -l, f(0,A)-[r,^]oo = 0, 9i(0,A) = 0, $(0, A) - [9, ^]oo = 

The immediate checking shows that for A € C+ such a solution is 

/ iei^t _i^giAC(g-iA<I.(f) ^ giAt) 

(6.55) v{t,X) = 

IgiAt _^^gaC(_^g-iA*(^) ^^giAt) 

Combining of (6.48) with (6.54) implies that the m-function of the problem (6.52), (6.53) is 

/^^o(0,A) go(0,A) \ ^^lr\K: 
"^^^'-[riOA) 3(0,A) + ij' ^eC\M. 
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Therefore by (6.55) one has 

™('^) = (^0 ^ J , A e C+. 

Applying the Stiehjes formula (6.23) to m(-) one obtains the spectral function of the bound- 
ary problem (6.52), (6.53): 

Since S(s) has the continuous derivative 

(6.57) ^'(^) = - 1 ^1 K 

it follows that i^(S; C^) is the set of all functions g{-) such that 

/ (S'(s)ff(s),ff(s))rfs<oo 

To simplify further considerations we pass to the new orthonormal basis {61,62,(33} in 
with ei = {^,0,--^}, 62 = {0,1,0} and eg = {■^75,0,^}. Then the Hilbert space 
L^(I) can be identified with the set of all Borel functions /(•) : I — >■ of the form 

fit) = fi{t)e, + h{t)e2 + Mt)e3 =: {fi{t), f2{t), Mt)}, 

where 5 5/1 € L^{I) and f2, h L^\^)- 
Next, the equality 

(l|-g-iA$(t)_|_gjAt^ \ 

e-*^* :C®C^C®C©C, AeC, 

defines the operator solution of Eq. (6.50) with i^(0,A) = ^^q"^- This and formula (6.16) 

(with </?[/(*, A) = ^{t,\)) imply that^ for each function /(•) = {h{-)j2{-),h{-)} € L\{T) 
the Fourier transform /(•) = {fi{ ), f2{ )} S L^(I];C^) is given by 

POO POO 

fi{s) = ^ I (e-*(*)^(i)/i(t) + e--V3(t)) dt, Ms) = e'^' h{t) dt. 

According to Theorem 6.13 Vf = / is a unitary operator from L\{I) onto L^(S;C^) and 
by using (6.37) one can easily prove that the inverse Fourier transform is 

= ^ / (e-^^*'*^/i(«) + ^,er''^^^'^-^^Us))ds, 

= ^ X^'""^*^''^-^^^"^ + ;^e-*/2(.))ds, 
/3(*) = ;;7f / + ;^e^'*(*+^)/2(s))ds 
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